Category Archives: lisp

Buffer Menu Sorting in Emacs 24

Upgrading to Emacs 24 a few months ago was mostly seamless. I only encountered one issue that was difficult to fix and has been lingering until today, when I finally got annoyed enough to devote some time to researching and fixing it.

I like having the Buffer Menu configured to always be sorted by filename. In Emacs 23, you could simply set the Buffer-menu-sort-column variable and you were done. That variable isn’t recognized in Emacs 24, despite claims to the contrary on various web sources.

If you look at the code in buff-menu.el, it was rewritten in 24 to use tabulated-list for its display mechanics. There is a Buffer-menu-sort function which is an alias to tabulated-list-sort. Once you are in the *Buffer List* buffer, you can call Buffer-menu-sort interactively using a prefix argument to specify the column: type ‘M-5 M-x Buffer-menu-sort’ to sort by filename. This works very nicely. Doing it a second time reverses the sort order.

So the challenge is to automate this, calling Buffer-menu-sort only once, automatically at the time a *Buffer List* buffer is first created. The mode will keep its entries sorted going forward.

I tried doing this by advising the list-buffer function, and having a buffer local variable keep track of whether I had already called Buffer-menu-sort once before. This didn’t work: something seemed to be resetting my buffer local variable, but I couldn’t figure out what.

The solution I ended up with is below. The advice function checks a variable called jc-buffer-menu which stores the current Buffer Menu buffer, and calls Buffer-menu-sort as needed. Put the code in your .emacs file, and you’re done.


string-match and dealing with state

(or, How I Spent a Few Hours Wrestling with Emacs)

I encountered some strange behavior and errors when calling replace-regexp-in-string in some Emacs code today.

My code was passing a function as the REP argument to replace-regexp-in-string. Here’s a similar, simplified example:

Fine, that’s what we expected. Now let’s change how mask-ssn works. It should still do the same thing, but does it?

Whoa, that’s not right at all!

What happened? Well, there’s a while loop in replace-regexp-in-string that calls string-match, then calls the REP function, and then uses the result in a call to replace-match. Since mask-ssn also calls string-match (via split-string), this messes up the next call to replace-match in the loop.

Figuring this out took a while, as my actual function was way more complicated than mask-ssn. I didn’t even suspect string-match to be the problem initially, so I went through the process of taking everything out and adding code back in line by line, chunk by chunk, call by call, until I pinned it down. In addition to weird string replacements, I also sometimes got an “args-out-of-range” error, depending on the size of the strings and the various functions using string-match that I experimented with calling.

There IS a solution, happily: save-match-data, which is better documented here than in the Emacs help. In a nutshell, the macro saves the current match state, evaluates the forms inside the body, and then restores the state afterwards.

That does the trick.

I discovered this only after going through the trouble of building a replacement to replace-regexp-in-string.

I felt pretty happy with myself after writing that. Since it doesn’t call any match functions after calling REP, all is well.

But then I thought about it some more. my-replace ITSELF is changing match data, so any callers will suffer from the same problem. D’OH. So we still need to wrap our code in save-match-data, if we’re going to be responsible.

I think save-match-data illustrates an interesting “pattern” of sorts. It deals with a problem of state, which is unavoidable, as we’re concerned with needing to track searches of text buffers. Since we HAVE to store this state, we can’t do it in a clean, purely functional way.

save-match-data handles this by saving the state of the string match in progress, probably pushing it on a stack somewhere (I didn’t dive too deep), letting the wrapped code manipulate the world, and then popping the original state off the stack to restore it afterwards. From the outside, it seems as if no state has changed, between the point at which you enter save-match-data and the point at which you exit. This allows for nested code, any number of call levels deep, to do what it needs, without interfering with other levels, as long as the relevant chunks of code are wrapped in the macro.

Pretty cool.

Composition and Inheritance

For as long as I can remember, writing any kind of non-trivial software meant you needed to use object-oriented programming. It was a no-brainer. So I learned all the fundamentals of OOP, and design patterns as well, since one couldn’t get very far in Java without knowing the most common patterns.

I think taking OOP for granted as the only natural way to manage complexity is why learning Lisp is so mind-blowing for programmers like myself. Take, for example, polymorphism. I didn’t know that there was anything besides parametric polymorphism–and I didn’t know it was called that; I knew it only as polymorphism, plain and simple. The ability of Lisp to do multiple dispatch was incredibly eye-opening.

I think this is the sort of thing people mean when they go on about how Lisp has broadened their horizons and deepened their understanding of concepts.

To mention another example, in a bit more detail: as an experiment in some recent python code for work, I’ve been using fewer classes/objects and more functions. (It’s debatable just how much actual “functional programming” mileage you can get out of Python, but I’ll put that aside for now.) In such an approach, you inevitably end up using a lot of composition, rather than object inheritance, to build higher level abstractions. And that’s been working out very well so far.

Composition is a powerful thing because you can control the granularity of code reuse. With a carefully constructed library of functions, you can choose to call the functions at the appropriate level of abstraction you need, and even mix and match. That’s much harder to do with object inheritance, where classes force you into an all-or-nothing package deal–if you want only some of the functionality, you need to instantiate the whole object anyway. And if you want to selectively override functionality, you need to subclass, which effectively ties the parent class to its subtree, making it harder to modify in the future.

I’ve been thinking lately that objects are useful mostly to facilitate data abstraction: there’s no reason not to group together related accessors and mutators. But when I consider more complex bundles of functionality, I think twice before creating a class hierarchy and see if I can do it using functional composition instead.

Exercise Mania

My ex-girlfriend’s mom started doing a jazzercise class in the 80s and hasn’t stopped since. It’s a weird thing.

Programming exercises can be just as addictive as some forms of well-marketed physical exercise. I have to actively resist writing solutions in Lisp to every silly problem I stumble upon online. But this morning, I came across this simple one (a blog post from 2007, but that happens on Hacker News) and, for some reason, couldn’t pass it up:

Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.

It’s so silly that I guess some folks, like the blogger linked above, have turned it into a problem of “what’s the most creative/strange way I can think of to do this?” Which is interesting and fun in its own right.

What I don’t understand is the attitude expressed in the original fizzbuzz blog post, “Using FizzBuzz to Find Developers who Grok Coding”. Does it really matter if it takes someone 5 or 20 minutes to write a solution? (FWIW, I took about 10 mins). A shorter time with such a goofy example means someone “groks” coding more than another? Ridiculous.

These types of puzzles are amusing precisely because a lot of real-world development (which is what the vast majority of people who code for a living do) is unfortunately pretty rote, and doesn’t require a lot of algorithmic thinking. Code monkeys learn to use libraries and frameworks, and spend a lot of time trying to use them correctly and cleanly, in order to implement straightforward requirements. So these exercises are a shift in mindset. In a tiny way, they put you back on the path of engineering. It’s why people find them interesting to do. It’s why I’m reading the Structure and Interpretation of Computer Programs.

Taking a few more minutes might mean you’re not accustomed to solving such problems every minute of your working life. But that’s not at all the same as gauging whether someone understands coding or not. That kind of measurement is naive at best.

An SICP exercise in implementing cons, car, and cdr

I started working my way through the Structure and Interpretation of Computer Programs this week.

Contemporary “intro to programming concepts” texts tend to focus on real-world examples in areas like business and web-based applications. In contrast, SICP involves a lot of math problems. I’m liking this about the book. Math is particularly good for illustrating the difference between declarative and imperative knowledge, between “describing properties of things and describing how to do things.” You may know, formally, what a square root is, but coming up with a procedure for calculating the square root of a number is another matter. Bridging these two kinds of knowledge is the essence of what computer programming is about.

The second section, “Building Abstractions with Data,” starts by discussing different ways to implement an abstraction for rational numbers. The text walks through using a Lisp cons cell to store a numerator and denominator pair, and using car and cdr as selectors. Then it provides this exercise, which I found particularly intriguing:

Exercise 2.5. Show that we can represent pairs of nonnegative integers using only numbers and arithmetic operations if we represent the pair a and b as the integer that is the product 2a3b. Give the corresponding definitions of the procedures cons, car, and cdr.

That’s a really clever way to store a pair! But perhaps I only think so because I’m not a math person.

At any rate, I wrote a solution in Common Lisp, with some helper lambdas to calculate log and pow arithmetically.

I’m finding that thinking about math problems and learning Lisp are helping me think more algorithmically about solving problems in the code I write for work. Maybe I’m starting to experience what Eric S Raymond famously wrote about Lisp: that it “will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot.” I’m holding out hope for that last part though…

SBCL 1.0.55 x64 needs GLIBC 2.14

It installed on my Ubuntu 11.10 (Oneiric) laptop, but when I tried to run it, I got this message:

sbcl: /lib/x86_64-linux-gnu/ version `GLIBC_2.14' not found (required by sbcl)

Hmmm. Let’s see..

# ls -al /lib/x86_64-linux-gnu/
lrwxrwxrwx 1 root root 12 2012-03-06 21:17 /lib/x86_64-linux-gnu/ ->


This has already been reported in the SBCL bug database.

My workaround was to grab 1.0.54 from the old file releases section on SBCL’s sourceforge site. That worked fine. So you’ll need to downgrade if you want to use the 64 bit version.

This came up when I ran into problems installing cl-libxml2 using quicklisp. The x86 version of SBCL throws an error when using cffi to load the Ubuntu-packaged libxml2 library, because it expects a 32-bit version of the library but finds a 64-bit one instead (I neglected to copy the error message, but it’s something about finding an unexpected/wrong class ELFCLASS64).

So I suspect anyone who has a x64 system that uses cffi to access libraries in this manner will run into the same problem with SBCL 1.0.55.

On a side note, I can’t imagine installing and managing Common Lisp libraries without quicklisp. It took a few seconds to do what would probably take a beginner like me hours to figure out. Awesome tools like that make a huge difference to newcomers who want to get up and running with CL without a lot of pain.

* * *

UPDATE: This is fixed in 1.0.56.

A Good Video Discussion on Clojure

The other day, I discovered this wonderful video from 2009. It’s an informal chat/whiteboard session with Rich Hickey, the creator of Clojure, on Microsoft’s Channel 9.

Watching it really helped solidify in my brain various concepts around Lisp and Clojure that I’ve been trying to learn recently in my spare time. I didn’t grasp every tangential reference to various debates in the world of languages and compilers (and there are many of them in this video!) but it’s still worth watching for the interested layperson who’s trying to delve into the world of functional programming.

The best part, for me, is Rich’s explanation of how data structures in Clojure are immutable yet very fast when it comes to copying. This seems counterintuitive, since, on the face of it, you would think that it’s very slow to have to copy whole data structures to make changes. The secret is that it doesn’t do a deep copy. The underlying implementation of lists and sequences uses shallow trees: “copies” of data only need to clone a short branch of the tree and reference a new root node, which also keeps the old data structure intact. The discussion is wonderfully elegant.

Having used only imperative languages my entire life, this recent foray into Lisps and functional programming has been really challenging and eye-opening. Putting aside its alleged strengths and advantages, I find the mathematical flavor of this style of coding very appealing. I don’t have a mathematically-oriented brain, but at the same time, oddly enough, something about Lisps seems truer to what I intuit the essence of computing to be. If I were formally trained in computational theory, I’m sure I’d have some fancy jargon to name this quality I’m trying to describe. As it stands, I can only say it feels close to the machine.

Making Emacs an IDE

It’s that time when bloggers wax introspective about the past year. For me, the major personal revelation in 2011 was re-discovering something very old, and putting it to new use. For me, 2011 was the year of the Emacs IDE.

I’ve been using Emacs, on and off, for close to a decade now. What’s changed is that, in the past few months, I’ve been writing extensions for it. It started with a simple desire to better navigate files in a complex directory hierarchy that followed specific and somewhat convoluted conventions. At first, learning Emacs Lisp was simply a means to an end, but I ended up liking it so much that I started exploring Common Lisp (and more recently, Clojure, since I’ve worked with Java in the past).

What started as a small task has become a larger project of turning Emacs into an IDE.

To understand this, one needs to know some context about the system I work with. We developers edit tons of XML files and files in other text formats, which all drive our proprietary web application product. We have many command line tools that manipulate these files in various ways; the system was originally designed by folks who followed the UNIX philosophy of building orthogonal tools and chaining them together.

There are pros and cons to this system; for reasons I won’t get into, I don’t love it, but it’s what we work with right now. When I started the job, the vast majority of the developers used screen, vi, and the shell prompt. Typical workflows that involved working with only a few files could be extremely hard to keep track of, and usually required a lot of copying and pasting between screen and/or ssh sessions. Few people seemed to mind, but I found the workflow to contain too much extraneous cognitive load, and the state of the tools made development very prone to error.

Gradually, I’ve been integrating our tools into Emacs. Sometimes that simply means binding a key combination to running a diagnostics program and storing the output in a buffer. Sometimes it means accumulating that output for history’s sake. Sometimes it means parsing program output, processing it in Emacs Lisp, and inserting a result into the current buffer. Sometimes it means running external programs, even GUI applications, and tweaking them a bit to tell Emacs to open specific files you want to look at.

The productivity gains have been amazing. This is no reason to brag: managing several screen sessions with different vi and shell instances wasn’t exactly hard to improve upon. But Emacs made it fairly painless. Emacs Lisp has proved to be wonderful “glue” for integrating existing tools in our environment.

Writing tools that enable you to do other job tasks better is a really interesting experience; I’ve never done it to such an extensive degree. So far, one other person in my group is using this Emacs IDE, and she has been happy with how much it facilitates her work. Others who swing by my desk for something often watch me work for a moment, and ask, “how did you do that?! that’s not vi, is it?”

Getting more people to switch over means convincing them that the steep learning curve of Emacs is worth the gains of the integration I’ve done. I’m not sure how much that will happen, since a big part of it is cultural. But if there aren’t any more converts, I don’t really care. The best thing about this ongoing project is that I am the end user. The software I wrote is really for myself. It is something I use intensively every single day. And that makes it all the more gratifying.

Getting the Solarized Theme to Work in Emacs

I discovered the Solarized color palette about a month ago, and have been using it since. I’ve found it helpful for reducing eye strain.

Getting the palette colors to accurately display in terminal-based Emacs under Ubuntu 11.10 took a bit of work. There’s a lot of scattered info about this topic if you google around. I first tried to do it with as little modification to my system as possible (I’m a minimalist), but I found that I absolutely had to do these three things to get it working.

1) Set the color palette in gnome-terminal. No matter what I tried, I could not get Emacs colors to look right without this step, so it’s a crucial one. Don’t skip it.

Solarized comes with an Xdefaults file, but that didn’t work for me. This blog post had instructions for setting gnome-terminal colors using gconftool-2, and that worked very nicely.

I put the commands in a handy little shell script that allows you to choose either the light or dark version of the palette. You can get it here:

2) Your TERM environment variable needs to be set to “xterm-256color.” The default “xterm” will cause the Solarized color theme to look completely crazy in emacs.

Add the following line to your ~/.bashrc line (if you already set TERM there or some elsewhere else, change it instead of adding this):

export TERM="xterm-256color"

3) Use this version of the Emacs color theme for Solarized:

Note that unless you’re using Emacs 24 (pre-release source code tarballs are available here), you’ll also need the color-theme package if you don’t already have it installed.

Put the files in a directory. In your .emacs file, you’ll need to add the directory to your load-path; I also added a bit of conditional logic for how to load the color theme in Emacs 23 and 24, since I switch between them:

(add-to-list 'load-path "~/.emacs.d/emacs-color-theme-solarized")
    (equal 0 (string-match "^24" emacs-version))
    ;; it's emacs24, so use built-in theme 
    (require 'solarized-dark-theme)
  ;; it's NOT emacs24, so use color-theme
    (require 'color-theme)
    (require 'color-theme-solarized)

That should do it.

Learning Lisp

For work projects, I often have to jump around a lot of XML files referenced in one another by name. The files follow a strict naming convention, but it’s one that looks like gibberish to human eyes. Typing them is difficult and prone to error; plus there are so many files in a given project that navigating a visual tree of the directory isn’t much easier.

So I wrote a small library for Emacs to find and open a file in the directory tree, if the cursor is over a string of text that matches a valid filename in the convention. I’m embarrassed to admit this took me two full days (a few other functions were in the library too). The only previous exposure to Emacs Lisp I had was tweaking my .emacs file, so I had to learn the language as well as hunt down the right functions to call for what I wanted. The work is paying off: it feels 10x’s easier to navigate project files now.

I experimented with trying to do this in Komodo Edit, a popular editor among coworkers, but it involved learning Mozilla’s XUL and Komodo’s own API, as well as writing javascript (yuck), so I abandoned that effort pretty quickly.

I didn’t think I would like Lisp, but the experience has been pretty fascinating, and I’m now making my way through the Practical Common Lisp book by Peter Seibel. It’s interesting learning a language that’s half a century old (!), and that’s influenced so many contemporary languages. One might think that there’s nothing there worth learning or revisiting, but that is so wrong. In particular, I’m trying to wrap my head around the power of Lisp macros and the way they allow you to create new syntactic abstractions. The idea of extending the very language itself, rather just adding new functions, is mind-boggling, to say the least. And, from what I understand, it remains fairly unique to Lisp in spite of the flood of new languages in the past few decades.

It’s disheartening not being able to find much info about who actually uses Lisp anymore, aside from hackers building modules for Emacs. Paul Graham has a cool essay, “Beating the Averages”, about using Lisp to build online store software that Yahoo eventually acquired. And ITA software, which makes an airfare search engine that powers the entire travel industry, uses Lisp. But aside from these bits of info, there isn’t much out there in the way of Lisp “success stories.”