Developing Lisp code on a free software platform is no mean feat, and documentation, though available, is dispersed and often too concise for users new to Lisp. In the second part of an accessible guide to this flexible language, self-confessed Lisp newbie Martin Howse assesses practical issues and implementations under GNU/Linux
________
Lisp is as esoteric as it is powerful and coding is truly the only route to mastering the intricacies of this complex and supremely intriguing language and the rapid development model presented by its REPL (Read Eval Print Loop) or top-level. Lisp masters well earn the title of wizards, and it's no coincidence that Abelson and Sussman refer to casting spells with Lisp, a common metaphor within the community and modified title of a popular beginner's tutorial at lisperati.com.
In the first article in this series, the conceptual underpinnings of Lisp were presented, giving a flavour of the language from a cultural and theoretical perspective. As an aside, before getting down to the real action of Lisp on a free software platform, it's also worth stressing the importance of Lisp to thinkers and mathematicians on the bleeding edge of theory, working artistically in what are often considered wholly scientific fields. Douglas Hofstadter, of GEB (Godel, Escher, Bach: An Eternal Golden Braid) fame provides an excellent introduction to the playfully recursive aspects of Lisp in his rich and rewarding Metamagical Themas. And both Bruno Marchal and Gregory Chaitin, working in the fields of consciousness theory and hardcore maths and computation respectively, have expressed huge interest in the elegance of Lisp. Aside from such lofty concerns which serve to fire the thirst for Lisp, what we're concerned with here is getting down and dirty with some simple application or example code, say from Paul Graham's excellent ANSI Common Lisp, and such an enterprise demands a robust and readily navigable working environment. We'll need to select from a plethora of free software Lisp (and dialects such as Scheme) implementations, choosing one which will readily suit a beginner, whilst still remaining open to others more suited to certain more demanding users and applications. Some free software Lisps can seem brittle and over-verbose is comparison to solid, hefty common tools such as gcc, gdb and associated autobuild suite. There's a lot to get used to here, and the implementation should be well bonded with an IDE such as GNU Emacs. SLIME is a contemporary solution which really does this job, and we'll check out some integration issues, and outline further sources of Emacs enlightenment. It's all about identifying best of breed components, outlining solutions to common problems and setting the new user on the right course, so as to promote further growth. And as users do develop, further questions inevitably crop up, questions which online documentation is poorly equipped to handle. Packages and packaging from both a user and developer perspective are often badly covered if it all. And whereas, in the world of C, everyday libraries are easy to identify, under Common Lisp this is far from the case. Efforts such as key SBCL (Steel Bank Common Lisp) developer and all round good Lisp guy Dan Barlow's cirCLe project, which aimed to neatly package implementation, IDE, documentation, libraries and packaging tools, would certainly make life easier for the newbie, but unfortunately progress does seem to have slowed on this front.
Many worlds
Without even throwing Scheme, a distinct dialect of Lisp, into the mix, and scarcely scratching the surface of smaller, more embeddable custom Lisps, the terrain of free implementations is vast and few maps exist to readily guide the newbie. Free Common Lisps include the exotically titled Armed Bear Lisp, CLISP, a pleasant interactive Lisp, GCL, the official GNU Common Lisp and the two big hitters CMUCL (CMU Common Lisp) and SBCL (Steel Bank Common Lisp). Other free Lisps of note include the cool and very powerful OpenMCL, which runs on a PPC architecture under LinuxPPC or Darwin, and in the latter instance includes bindings to Cocoa and Carbon, which makes for elegant application building. Various comparative surveys have been made of these major free Lisps, with the ubiquitous cliki.net resource providing at-a-glance tables surveying the terrain as to features, ANSI compliance and performance across a good range of benchmarks. Of course compliance is important, particularly if you're interested in coding portable applications across implementations, but there are plenty for other issues which effect the choice, particularly for new users. Once Lisp experience has been gained, other flavours can be checked out as it's obvious that some Lisps are better suited to particular projects.
Alongside portability, other pressing issues include the level of integration with a suitably powerful IDE. In this instance SLIME (Superior Lisp Interaction Mode for Emacs) has been chosen as best of breed component which fits the bill here, and CMUCL, SBCL, and OpenMCL all play well here, with work in progress on CLISP. Other key issues in assessing Lisps include the availability of functional libraries, delivery of executables and the nature of compilation. The last is a tricky topic which involves some digging deep into how such Lisps work. However although such technical issues do have some bearing, choosing an appropriate Lisp is a lot more about style and flavour, with factors such as verbosity and ease of re-entering the REPL or top-level as major elements here.
Under GNU/Linux on an x86 architecture, CMUCL and SBCL really are the major players, garnering most respect from the community. It's also worth mentioning that SBCL now offers 64 bit support for the AMD64 and EM64T platforms. CLISP is well worth keeping in reserve, however, with good performance for some tasks, and readline providing an excellent level of command line interaction. It's a good, lightweight Lisp to fire up at odd moments, when you simply don't need the burden of a fully fledged IDE. CLISP can even be used as a shell for those who like to see Lisp everywhere.
Given that CMUCL and SBCL can readily be viewed as brothers, forking from the same stock or code base, the choice between these high performance Lisps is something of a tough one, boiling down to look and feel, and compatibility with various libraries or research apps such as Common Music. Indeed, SBCL derives so much code from CMUCL, created at Carnegie Mellon University, that save for the confusion such a naming would cause, CMUCL has been mooted as an equally appropriate name for this vibrant offshoot. In practise, SBCL's rather ugly expansion into Steel Bank Common Lisp is an acknowledgement of the industries where Andrew Carnegie and Andrew Mellon, respectively, made the money. The name also well refers to the heavyweight, industry-strength qualities of this powerful Lisp, and though favoured by hardcore free software hackers on the Lisp scene, such as key developer Dan Barlow, SBCL lives up to the tough, heavy handed, somewhat ugly inspiration of its name. That said, SBCL does expose a decent feature set with native threads as a major bonus, but most differences between the two siblings lie under the bonnet and relate to the differing development models of both Lisps.
Steel pulse
SBCL stresses standards compliance and maintainability through cleaner concepts and code, and perhaps as a result does appear to be undergoing more active development with ports across a vast range of architectures. Bootstrapping has been simplified, but the major difference lies within the compiled or interpreted nature of each Lisp. On the surface both present the standard top-level interpreter which can be used interactively, but SBCL does offer a simpler more or less compiler-only model. In the case of CMUCL things are more complex, not to say arcane, though even digging around the internals with Common Lisp functions such as compiled-function-p, which tells us if a function has been compiled, in nearly all instances shows few differences. CMUCL offers two levels of so-called interpretation, alongside both byte-code, for a VM (virtual machine), and native or machine code compilation. The lowest level presents a tiny baby interpreter which handles the really basic stuff like value assignment. Next up is the fully fledged interpreter which makes use of the first stages of the byte-code compiler to convert code to a suitable internal representation. It really behaves more like a compiler, handling expansion of all macros. These interpreters are invoked from the interactive top-level, or when code is loaded as source. Compilation is explicitly commanded with functions such as compile-file. SBCL takes a similar approach, but cuts out the middleman. In both instances, the speed of a compiled language is proffered with all the development benefits of interactive interpretation. It's a win win situation. In common with CMUCL, SBCL offers a low level, basic interpreter for easy forms. More complex stuff is thrown straight at the native compiler. Of course it's all still totally interactive, and compiled-function-p is our only clue that we're not dealing strictly with an interpreter.
SBCL does present a more attractive package given its more active
development model, which means that libraries and build systems are
well packaged for it. The ASDF package management system, though now
ported to CMUCL, was originally an SBCL only affair, and packaging of
asdf-install does reflect this. Sockets are also easier to get to
grips with under SBCL, and there's a good deal of documentation
regarding the installation and configuration of most libraries under
SBCL. On the CMUCL side, graphical support is more stable, with a
stable cl-sdl library providing SDL (Simple DirectMedia Layer) support
and excellent OpenGL bindings. CLX, the Common Lisp Xlib, is also more
mature under CMUCL. For the beginner CMUCL presents a serious option,
running in a friendlier manner out of the box and easily integrating
with SLIME. Output is less verbose and everything works well. It's
quite simply a softer landing for the newbie, and presents fewer
problems with apps such as Common Lisp Music and Common Music which
present high-level Lisp-based compositional and sound processing
environments. For the more audio inclined both apps offer exciting
opportunities for experimentation, alongside an easy platform for
learning Lisp. Installation of CMUCL and associated packages is a
simple enough affair under either Gentoo or any apt-enabled
distribution. It's worth mentioning that most Lispers do prefer a
Debian-based distro, and many online Howtos are dedicated to this
flavour.
Without digging too deep into Scheme, a small and conceptually
pleasant Lisp alternative, PLT Scheme and associated DrScheme
environment present a supremely usable coding platform. Other Lisps of
interest, which fulfil particular functionalities in relation to
extending or embedding code include SIOD (Scheme In One Defun) and
Guile on the Scheme side, and powerful librep, on the Lisp
case. However before considering any of these as scripting or
extension language for a large scale app, it's worth checking another
approach to embedding which would make use of Lisp's native FFI, or
Foreign Function Interface, which can be used to link to any C library
from Lisp code. That said, if you do have a hefty body of C code which
readily needs a scripting or interactive component, librep is a
powerful solution which has been successfully adopted by apps such as
the Sawfish window manager and Jade editor. And, of course, what could
be a better exercise in learning Lisp internals than rolling your own
interpreter in a language of choice. There's plenty of material out
there to assist in such a task, with the work of Abelson and Sussman
once again very much to the fore.
EINE, ZWEI
Lisp can well be developed following the edit-compile-test cycle of languages such as C, with, for example, an editor providing indentation, highlighting and shell access, and any of the Lisps under consideration compiling to native code for debugging. However, as we've seen in the last issue, the beauty of Lisp lies in supreme interaction, with on-the-fly bugfixes a distinct possibility. The REPL provides for this interaction in theory, but in practise an IDE which integrates with this functionality and extends it radically is very much in order for this complex language. Such an IDE should free the coder from banal concerns with syntax and numerous shortcuts should ease repetitive actions and provide for flexible, customised development models. Though offering highlighting and formatting, a good IDE is much more than a text editor, and should allow programmers to examine and engineer their code at a range of levels. And the very nadir of IDEs, not to say the greatest programming environment ever, was to be found bundled with the original Lisp Machines, an important milestone in the history of Lisp and an area well worth rewarding research for the Lisp newbie. Seasoned Lispers come over all dewy eyed at the mere mention of Genera, the OS that ran on the Symbolics Lisp Machines and which included an integrated editor, excellent GUI and a range of functionalities to examine objects and processes. Genera is hard to beat when it comes down to integration and it's tough to give an idea of the flexibility afforded by the tight bonds between editor, object oriented programming language, GUI toolkit (CLIM) and the OS itself. Genera occupies an important position both within the history of coding and Lisp itself and it's worth noting the appearance of the Emacs family in this field. With customary hacker wit, and echoing the recursion wrapped up in the GNU acronym, the original Lisp Machine's Emacs implementation was called EINE, standing for 'EINE Is Not EMACS'. And more cleverly a second version of EINE on the Lisp Machines was known as ZWEI, for 'ZWEI Was Eine Initially'. GNU Emacs, on which we'll build the contemporary IDE of choice, grew from the original Multics implementation, and though thoroughly extendible using its own dialect of Lisp, elisp, is written primarily in C. Indeed, GNU Emacs is described in part by the GNU project as an elisp interpreter with extensions to support text editing. For the sake of completeness, XEmacs, a fork of GNU Emacs which presents much of the same functionality, is worth mentioning but will not be examined in any detail.
Superior Interaction
Emacs, and particularly GNU Emacs, has spawned a rich culture, or perhaps religion, which views Emacs as a way of life. There's a fairly steep learning curve but built-in tutorials should take some of the burden and the two excellent O'Reilly works covering GNU Emacs use and customisation are invaluable for those seeking to get the most from this radical tool. Online cheat-sheets and .emacs configuration files are readily available to suit all needs, and once basic text editing has been mastered, the full IDE can readily be assembled by throwing SLIME into the mix. SLIME (reviewed in issue 45) pushes Emacs as IDE further towards the Genera model, with a good deal of low level functionality for debugging purposes, cross-referencing of function callers and variables, access to online documentation and a host of completion, indentation and parentheses functions all available with a range of Lisp REPLs. Installation is a simple enough affair and there are a good many Howtos online covering basic use. SLIME plays well with CMUCL and SBCL, with CMUCL perhaps providing a cleaner experience. Help for key bindings for the SLIME mode are readily accessible through the usual Emacs functions, and documentation touches on nearly all areas of SLIME use. With such bindings mastered, and key concepts understood, SLIME really does take care of the gnarly bits for novice coders, and can well function with code examples as a good teaching aid. Beyond the SLIME basics which are well described in both issue 45 and online, with cliki.net again serving as central resource, SLIME also features a decent Emacs-integrated debugger and offers good levels of customisation, using the standard Emacs means.
Following the continuing rise and rise of SLIME, few would choose to deal with ilisp, the rather unfortunately titled inferior Common Lisp interface which ships with GNU Emacs. Ilisp fulfils some of the basic functionality of SLIME in connecting Emacs to many Lisps, but is brittle in practise and doesn't come close to SLIME's feature set and flexibility. Emacs own Lisp mode can be used aside from extending Lisp to check out basic code, though it's worth remembering that the elisp dialect differs in major ways from Common Lisp, and newbies may well slip up on its unusual dynamic scoping. Other IDEs or editors of note include Hemlock, CMUCL's own native Emacs-alike editor, which has inspired the elegantly titled Portable Hemlock, described in the words of the developers as "An attempt to free Hemlock from its CMUCL prison." On the GUI side, Jabberwocky is a major player which supports all of our free Common Lisp implementations including CLISP and boasts a decent editor, with the usual fontification, parentheses matching and indentation at work, debugger, project explorer and self-styled interaction buffer. Only one drawback for Lisp purists though; the GUI is pure Java, necessitating the unfortunate installation of a JDK or JRE.
Best of breed
There's certainly a good many useful libraries which can be installed with supreme ease using well-built packaging tools such as ASDF or Common Lisp Controller. Again, it's all about identifying best of breed components which should see the novice Lisp programmer over most everyday hurdles. CLX and cl-sdl have already been encountered within a graphical context, and bundled examples should make clear usage. Cl-sdl is more in the order of a set of bindings or FFI (Foreign Function Interface), and in this context it's well worth mentioning UFFI, a library which provides an extra portable layer on top of FFI. McCLIM presents a more high-level GUI toolkit, and can readily be viewed as a free and portable implementation of CLIM, our elegant old friend from the days of Lisp Machines. Garnet, originally developed at Carnegie Mellon, presents high level abstractions for GUI design. Expanding the acronym gives some flavour of what to expect from this complex toolkit; Garnet stands for Generating an Amalgam of Real-time, Novel Editors and Toolkits.
The networking side of things does present more of a minefield, with some divisions as to Lisp implementation. Sb-bsd-sockets provides for a complete BSD sockets API, but runs only with SBCL, and the older db-sockets which would run with CMUCL is currently unsupported. Trivial-http and trivial-sockets can satisfy basic networking needs across most of the implementations covered, and a huge range of further libraries cover specific protocols such as LDAP, SNMP and SMTP. Though nowhere near approaching the wealth of libraries available under languages such as C, Common Lisp does have most bases covered, with support for a range of databases, regular expressions, XML, and encryption. Using UFFI or an implementation's native FFI, and following example code such as the cl-sdl library itself, it's relatively easy to access C library functionalities from Lisp code. With libraries rounding out the powerful picture of a free, practical, and complete Common Lisp environment on GNU/Linux, it's clear that Lisp really does present a brave new choice for future coders intrigued by the wizardry which can be achieved with such a concise, expressive and elegant language.
Wrapping it up
Dealing with larger projects through breaking down code into multiple modules or packages is a brutal affair in C or C++, with sacrifices to modularity in some instances, but at least it's always been clear how to achieve this partitioning and well rehearsed tools and build systems do exist. Lisp newbies may not immediately need to know how to work on large or collaborative projects, but many do feel the lack of this knowledge, and building other's projects can be a confusing business the first time round. Packaging and namespace are the watchwords when it comes to large-scale Lisp modularity, and the relevant Common Lisp expressions reflect the former with in-package, defpackage and use-package as major players here. Packages provide for decent modularity, preserving separate namespaces for symbols. A file containing a distinct package of code would commence with a defpackage which defines the new package, specifies which other packages will be used and which symbols will be exported and can thus be used without a qualifier. A subsequent in-package makes of the following code the defined package. Use-package allows us to access all of the exported symbols, or alternatively import can be used for specific symbols.
Packages are well documented as to their theoretical use, and once the namespace notion is well understood, are relatively easy to use, but as usual some practical issues are overlooked. For example, package definitions will need to be loaded or defined before they can be referred to. Load will thus be used to provide appropriate pathnames and Lisp code can easily be written to deal with things in the right order. Peter Seibel's excellent Practical Common Lisp, freely available online and soon to be published in print, provides the best tutorials on package gotchas. Larger projects may well need to make use of mk-defsystem, the Lisp equivalent of make, and ASDF (Another System Definition Facility), an excellent, if tersely documented, system which can even automate downloading of relevant source. ASDF is bundled as part of SBCL, or it can be obtained with the Common Lisp Controller, which further automates the installation of libraries and source on Debian and Gentoo systems. Again, cliki.net is probably the main provider of documentation, and indeed itself functions as a clear component within the ASDF system.
key links
Casting SPELs In Lisp: http://www.lisperati.com
Paul Graham: http://www.paulgraham.com
SBCL: http://sbcl.sourceforge.net
CMUCL: http://www.cons.org/cmucl
SLIME: http://common-lisp.net/project/slime
CLISP: http://clisp.cons.org
OpenMCL: http://openmcl.clozure.com
SIOD: http://www.cs.indiana.edu/scheme-repository/imp/siod.html
Guile: http://www.gnu.org/software/guile/guile.html
Librep: http://librep.sourceforge.net
PLT Scheme: http://www.plt-scheme.org
CLiki: http://www.cliki.net/index
Successful Lisp: http://www.psg.com/~dlamkins/sl/contents.html
Practical Common Lisp: http://www.gigamonkeys.com/book
Common Lisp Cookbook: http://cl-cookbook.sourceforge.net