Clojure was designed to be a general-purpose, practical functional language, suitable for use by professionals wherever its host language, e.g., Java, would be. Initially designed in 2005 and released in 2007, Clojure is a dialect of Lisp, but is not a direct descendant of any prior Lisp. It complements programming with pure functions of immutable data with concurrency-safe state management constructs that support writing correct multithreaded programs without the complexity of mutex locks. Clojure is intentionally hosted, in that it compiles to and runs on the runtime of another language, such as the JVM. This is more than an implementation strategy; numerous features ensure that programs written in Clojure can leverage and interoperate with the libraries of the host language directly and efficiently. In spite of combining two(at the time) rather unpopular ideas, functional programming and Lisp, Clojure has since seen adoption in industries as diverse as finance, climate science, retail, databases, analytics, publishing, healthcare, advertising and genomics, and by consultancies and startups worldwide, much tothecareer-altering surprise of its author. Most of the ideas in Clojure were not novel, but their combination puts Clojure in a unique spot in language design (functional, hosted, Lisp). This paper recounts the motivation behind the initial development of Clojure and the rationale for various design decisions and language constructs. It then covers its evolution subsequent to release and adoption.
is a Lisp, but what does that mean? Lisp is a fascinating set of ideas, a family of languages without a monotonic lineage [Steele and Gabriel 1996]. There are some shared characteristics, all of which I considered positive attributes to retain for Clojure: code is data; independent read/print; small core; runtime tangibility; the REPL; and extreme flexibility. 3.1.1 Code is Data. A Lisp program is defined in terms of the interpretation of its core data structures, rather than by a syntax over characters. This makes it easy for programs to be a source of programs and for program transformations to be functions of data → data, and in both cases for the standard library to be of substantial utility. The conversion of character streams into data is handled by separate functions of the reader. While much is made of these facilities as a recipe for syntactic abstraction (e.g., macros), syntactic abstraction via code-is-data is just a single exemplar of the value of defining any process as functions of data → data wherein the standard data structures and library are of significant utility and reusability. This is in stark contrast to the common tendency to design a bespoke interface and types for every problem
Lisp Choices Clojure never intended compatibility with a predecessor Lisp like Scheme or Common Lisp, which left me with the task of deciding which way to go on several core Lisp decision points. 3.2.1 Lisp-1 and Macros. Clojure uses a single resolution mechanism for symbols in the function position and elsewhere, and is thus a Lisp-1 [Gabriel and Pitman 1988]. This is strictly less complex than Lisp-2, more conducive to programming with higher-order functions, less context dependent etc. The only exceptions to this are some of the symbols designating host interop accepted only in the function position, for which there are no meaningful corresponding values. However, Clojure supports programmatic macros (arbitrary functions of data → data, such data including symbols) in the Common Lisp style, which has been considered to be problematic Proc. ACM Program. Lang., Vol. 4, No. HOPL, Article 71. Publication date: June 2020. 71:8 Rich Hickey without the separate function space of Lisp-2. I concluded that in Common Lisp the conflation of symbols and storage locations, and the interning reader, were the sources of the conflict. The interning reader is particularly problematic; it creates a stateful interaction between reading and the runtime environment, breaking the otherwise functional nature of code-text → data-structures. By manipulating the environment, the interning reader restricts and interferes with the macro system’s ability to control evaluation. In Clojure, symbols are a simple immutable type akin to strings, with no storage. While symbols have two parts, an optional ‘namespace’ part and a ‘name’ part, they are not inherently interned in packages/namespaces. During read there is no interning nor resolution of symbols. The resolution of symbols to vars (Clojure’s separate storage location type) generally happens during compilation. Asecond place where resolution of symbols occurs is in Clojure’s version of quasiquote, called syntax-quote and designated by ‘`’. Syntax-quote will take a form containing symbols and replace any unqualified symbol (i.e. one without a namespace part) with a fully-qualified one, where the namespace portion corresponds to the mapping in the namespace where syntax-quote was called. Thus a macro’s use of the symbol rest (a Clojure core function) within a syntax-quote will be replaced with clojure.core/rest and will be free of conflict when that macro is expanded in a context with a different or local binding for rest. If one wants to introduce conflict-free local names Clojure has gensym. Taken together, this has worked out to be a quite satisfactory recipe for avoiding the so-called “hygiene problem” [Kohlbecker et al. 1986] 3.2.2 Reader Macros. One thing conspicuously missing from Clojure are Common Lisp’s reader macros. While they fully empower the user to create a language of their own specification, such languages are islands. Critically, they require the presence of specific code to read particular data, such requirement being in direct conflict with some of the benefits of independent read/print enumerated above. I saw reader macros in the CL style as being in direct conflict with library development, interoperability, Clojure data (edn) as a wire format etc. I am certain that had Clojure had reader macros, and users availed themselves of them, the large, composable, data-driven library ecosystem that arose around Clojure would have been compromised or thwarted. I am not opposed to extensibility, and added it to Clojure’s data format with its christening as edn (the e stands for extensible) in the 1.4 release in 2012, described in section 4.5. 3.2.3 Tail Calls and Recursion. I very much would have liked to copy Scheme’s ‘proper tail recur sion’ [Steele Jr and Sussman 1978] and the elegant programming style it supports, but the JVM does not allow for that stack management approach in any practical way. On the other hand, I consider partial tail call optimization, e.g., of self-calls but not mutually recursive calls, to be a semantic non-feature. In addition, even with proper tail recursion, I thought it may be a point of confusion among users when exactly a call is in tail position and subject to the optimization. Thus, while supporting recursive and mutually recursive function calls (without Scheme’s unbounded promise), Clojure offers a limited but explicit looping recursion construct called recur and an accompanying binding/scope construct called loop. recur takes arguments matching the bindings in the enclosing loop, or the function head if no enclosing loop.
CoreDifferences Setting aside being hosted, the above differences with other Lisps were, I thought, insufficient to justify making another Lisp dialect from scratch vs building one atop Scheme or Common Lisp. There are several key areas where Clojure differs at its core from Scheme and Common Lisp and thereby justifies its existence. 3.3.1 Abstractions at the Bottom. Due to their history and heritage, Lisps prior to Clojure, and their standard libraries, have been built on concretions: primarily the mutable cons cell and lists constructed thereupon. While one can’t fault the choice during the early days of Lisp, we now have sufficient technology (high performance runtime polymorphic dispatch, thanks OO!) to support different choices about what should be at the bottom. One positive aspect of years of OO programming is that it left me with a strong aversion to defining libraries in terms of concretions. I simply couldn’t imagine choosing a cons cell with mutable car and cdr as the basis for a language or library in 2005, and so I did not. It was my objective to tweak Perlis [1982] “It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures.” with “It is better to have 100 functions operate on one data abstraction than to have 10 functions operate on 10 data structures.” For Clojure I built several such abstractions underlying its standard library covering sequentiality, collections, keyed access, etc. The abstraction replacing the cons cell is seq.