Lisp for Spotlight

Richard Cook recently posted code for a Spotlight plugin to index Lisp files. His plugin takes what's probably the absolute simplest approach: Indexing a Lisp source file as if it were plain text.

When I saw his post I played around with Spotlight a little. I noticed that at least some, but not all of my Lisp files were already being indexed. It turned out that the ones that were being indexed were some pretty old files, ones that I'd created or edited in MCL. They had an HFS+ file type attribute (yeah, file types and creator codes still exist in HFS+) of TEXT, which seemed to lead to their kMDItemContentType being com.apple.traditional-mac-plain-text, which meant that they get indexed by one of the standard Spotlight plugins.

It seemed like there must be some easy way to convince OS X that all files with a .lisp extension are text files that can be indexed like any other text file without going to the trouble of actually writing an indexing plugin, but Pierre Mai did some experiments along those lines that didn't pan out.

I'm not convinced that there isn't some tweaky little thing that might be preventing Pierre's attempt from succeeding, but maybe there are other easy hacks. For example, a simple modification of the existing SourceCode.mdimporter or RichText.mdimporter to convince them to index Lisp files.

But who wants dumb plaintext indexing, anyway? We should be able to do better. Function definitions, class definitions, who-calls info, package definitions, ASDF or other system definitions—There is lots of potentially useful stuff we could index.

Jonathan Wight's Spotlight plugin for Python should be an inspiration. It indexes file-level comments, author information, version, defined functions and defined classes, so you can do queries like this:

# Find all files containing a function called "main" and a class called "Foobar"
$ mdfind "org_python_functions == 'main' and org_python_classes == 'Foobar'"

It also hooks into the Spotlight GUI:

As well as the Finder:

Appealingly, the Python plugin uses the built-in Python code walker to parse files. A more static approach might be the way to go for Lisp (Lisp code walkers seem to get so nasty), in the style of the old approximation of looking for lines that begin with “(def” to find function, class, variables, etc. definitions.

Get to it, Lazy Web!

Update: The lazy web, she is not reliable. I had to do it myself.

darcs trackdown

Oscar Ruiz is a helicopter pilot in Mexico City, and a photographer. Also see this, this and this. Via things magazine.

In the realm of source code repository visualizations, Christian Neukirchen graphs the size of a darcs project over time in terms of lines of code.

The idea behind darcs trackdown is cool, I wish I'd thought of it.

The following projects all, weirdly, generate graphs of the number of lines of code over a project's lifetime (some do a little more than that): statcvs, cvsplot, bloof. From Matt Doar's comment on a previous post.

Sunset Junction 2005

Ahh, people. Photo by Lori.

Monday morning means no more rolling out of bed right into the brutal LA sun and walking a couple hundred feet to get a bowl of Hawaiian BBQ and a margarita. And it saddens me.

That's right, Sunset Junction is over. And JoAnne didn't get punched out, only slapped.

Ahh, JoAnne. Photo by Lori.

I have some pictures. Lori has some better pictures.

Previous Sunset Junction posts: 2002, 2003, 2004. And it's not right unless I link to Tony Pierce (now with photo essay).

Source Code Respository Behavior

A recent paper by Rob Pike on a new system for automating analysis of very large data sets (using their new language, Sawzall) included a graph that I thought was kind of cool. It showed a histogram of commits to the Google source code repository by minute of the week:

There is some obvious structure: Weekdays are busy, weekends less so. There's definitely a dip around lunchtime and Friday is the least “productive” day of the week, if you measure productivity by code commits. There is a spike just after midnight every day—maybe some automatic process is committing code? Auto-syncing of repositories?

I became curious about the idea of repository visualization. I was curious to see the histogram for the Evolution Robotics repository, so I wrote some code to do some very simple analysis of a cvs log. I actually executed “cvs log” on the top directory and counted every line that looked like this as a commit:

  date: 2005/06/01 17:23:59;  author: cody; [extra stuff I ignored]

I can't show you the result of that analysis because the repository isn't public. I won't show you the code because it's ugly. So what's left?

I can run the same code on SBCL's repository at SourceForge. Based on advice from SBCL developers on #lisp, I ran cvs log on the file version.lisp-expr instead of the top level sbcl directory, to more closely approximate measuring something like significant commits. Here's the resulting SBCL histogram:

SBCL commit histogram, by hour of the week. Rendered by gnuplot. Click for the PDF version.

I was a little surprised that the valleys in SBCL activity correspond pretty closely to midnight California time. The valleys are, however, much narrower than the ones I observed in the histogram associated with a repository mostly worked on by full-time employeees who all live in the same time zone (and they seem narrower than the google valleys, too).

Next I became interested in seeing the relative activity of the various committers to a repository. I wrote some code to keep track of the number of commits made each month by anyone who ever made a commit. I wrote more code that used CL-PDF (with Zach Beane's patches) to generate charts in the style used by babynamewizard.com's Name Voyager Java applet, which is a pretty nice way to visualize the relative popularity of various baby names over the past 120 years using census data. In my version, each committer gets a band of color whose vertical thickness at any given point on the X axis is proportional to the number of commits made by that person in that month.

SBCL committer trends, by month. Rendered by CL-PDF. Click for the PDF version.

The colors are chosen at random, so it can be a little hard to distinguish one person from another, but it's clear that for about the first 16 months or so wnewman was the sole committer, until he was joined by crhodes and dan_b. About six months after that, adejneka starts making commits.

One feature I half-expected to see in these trend charts was a wide range of committer “productivity”, perhaps even spotting one of the legendary superhackers who makes 10 (100!) times as many commits as the rest of the team. I think there are enough damping factors to keep the ratios from getting that crazy; There were, however, some people who were consistently two to three times as active as the rest of the developers. And from working with them, I know they actually were at least that much more productive than other programmers and the disparity wasn't just an artifact.

I would be hesitant to look at these charts for anything more than entertainment, though one might imagine that something like this would be included in a kind of anthropological analysis of software development. Or Taylorism.

Fun fact: The highest resolution version of these charts, the vector-based PDFs, are much smaller files than the high resolution PNGs I created as an intermediate step, which were much smaller than the low-res JPEGs I was originally going to post (but found lacking in quality) as well as the low-res PNGs you see here.

TabEdit

This lisp porn is practically medieval.

TabEdit is a Climacs-based lute tablature editor, developed as part of the Electronic Corpus of Lute Music (ECOLM) project at City University London's Centre for Computational Creativity.

This poster gives an overview of TabEdit and the system it fits into.

The TabEdit application is fundamentally a text editor, augmented by syntax analysis facilities. As in other text editors such as Emacs or Notepad, the user types in text: there is no constraint that the contents must always be syntactically valid. However, the text buffer is incrementally parsed at each keystroke (in constant time for the typical case), and any syntax errors are presented to the user.

The implementation “uses climacs to parse a textual representation of tablature and display the graphical output, which is mouse-sensitive (thanks to mcclim) and provides easy commands for interaction with the textual encoding”, according to Christophe Rhodes. The implementation is discussed further in “Syntax Analysis in the Climacs Text Editor” by Christophe Rhodes, Robert Strandh and Brian Mastenbrook.

First Autonomous Ground Robots In A Combat Zone To Be Deployed In Iraq?

Northrop Grumman is sending robots to Iraq, which it says will be the first autonomous ground robots in a combat zone.

From the Los Angeles Times:

Hoping to reduce human casualties in Iraq, the Army is preparing to use unmanned robotic ground vehicles built by Northrop Grumman Corp. for use in surveillance and border-security missions, the company said Monday.

It would be the first use of autonomous ground vehicles in a combat zone, the Century City-based defense contractor said

The company's Tactical Amphibious Ground Support vehicle weighs 3,400 pounds, can reach a speed of 25 miles per hour and is about the size of a compact car, with a flatbed top and tank-like treads that can be interchanged with tires.

[...]

But Nisbet cautioned that the Northrop robots' effectiveness still had to be tested, especially in difficult terrain.

The tactical amphibious vehicle can travel autonomously between set points on a path that's punched into a computer map by a nearby soldier, the company said. If it meets an obstacle, it can call for help from the operator, who can also take over full manual control of the vehicle.

Northrop Grumman has absolutely no information on TAGS on their website. Neither does the Northrop subsidiary that supposedly developed the robot, Remotec (jesus, people; you're defense contractors, can't you afford a real website?). None of the press releases mentions a company called Autonomous Solutions, but they have a platform they call TAGS (though they say it stands for “Tactical Autonomous Ground System”). It's clear from the photo in this article that the Northrop Grumman/Remotec TAGS is actually the Autonomous Solutions TAGS:

Kiwi

click for the boobs

Via Rainer Joswig, there's a new Lisp Wiki system in town. Kiwi was based on CLiki, though “very little of the original Cliki code remains.”

The current version should have feature parity with Cliki and the following new user-visible features:

• Page namespaces including a namespace for per page discussion pages like Mediawiki.
• More thorough RSS support.
• Viewable page histories.
• Better search functionality.
• CAPTCHAs to help prevent spam.
• The ability to preview edits before saving.

We believe that the new architecture will make it rather straight-forward to add other features such has expanded RSS support, a non-HTML “wikitext” syntax based on MediaWiki's, and a system for creating pages that can be accessed only by registered users

Sounds awesome, I love it. Too bad the source isn't available yet. And it's not clear who wrote it—Carl Shapiro?

P.S. Kiwi is powering the new ALU wiki.

Kent Pitman in Boston

Kent Pitman is going to be at the next Boston area Lisp Users' Group meeting [content-free link].

A meeting of Common Lisp users will be held at MaK Technologies location in Cambridge.

A short talk is planned Kent Pitman, who will speak on “The State of Lisp” followed by a round table discussion.

Why does their email point to a url with no info?

2,857,120 Combined Pixels of Power

I feel all studly with my new dual display setup, and my homemade laptop stand (thanks, extruded aluminum!). But hey, this is where I make my living now.

OOPSLA Promotion

Wallpaper design by William Morris, 1896.

OOPSLA has an interesting promotion this year, with a prize offered to the person who refers the most people to sign up for the conference [via gavin].

Have you ever wondered if the committee members stay in rooms different from yours? Well, in some cases, you would be right. The winner of the Promote OOPSLA contest gets to stay in the suite reserved this year for the Chair of the Program Committee, Richard P. Gabriel.

Cool!

And Dick will be staying in a regular room instead.

Aww.

White Stripes at the Greek

I saw the White Stripes last night with Mark and Charley at the Greek Theatre. It was OK. They could have been louder. And somehow we missed the Greenhornes.

The Drawing Room was nice afterward.

CL-GODB

Marco Antoniotti announced that CL-GODB, a Summer of Code project that allows one to access the Gene Ontology Database from Lisp, has been officially released.

The Gene Ontology (GO), is a controlled vocabulary used to aid the description of gene and gene product attributes in a variety of organisms. There are three different ontologies that describe molecular functions, biological processes, and cellular components.

The Gene Ontology Database (GODB) stores the known information about these genes and gene products.

I think this is the first Summer of Lisp project to reach this milestone.

Five Approaches to S-Expression Highlighting

paste.lisp.org's popular paste service for IRC users has fancy Javascript CSS to color code matching parentheses as you mouse over expressions. Unobtrusive, but not really all that helpful, either. B-.

The Community-Scheme-Wiki has mouseover Javascript CSS magic, but highlights entire expressions (including whitespace). Helpful, but the coloring of the indenting whitespace looks weird. C.

foom's emacs hack focuses on indentation. Clear and harmonious, B+.

michaelw's emacs hack is a hybrid of indentation-based and nested boxes, and removes parens completely. Deleting parens is bold, but... too bold. And grayscale looks techy but cold. C+.

pkhuong's emacs hack is a rainbow of expression-coloring, minus the whitespace. Nice try, that changing the hue according to an expression's position in its parent expression thing, but in the end it is most valuable for what we learn from its failure. D.

(All examples culled from #lisp.)

(Nobody using pastel colors gets an A.)

New Age of the ALU

Duane Rettig offers up more information about the ALU in a single paragraph than I think I've seen in years:

The ILC2003 conference was our first attempt to run a conference ourselves.  In many respects, it appeared to be a success.  In other ways, it was a failure that almost turned to disaster, as those of you who tried to register at the hotel that Saturday may have discovered.  Some fast thinking people, including Heow Eide-Goodman (now a board member, and I believe principal organizer of the Lispnyk group) and Lilliana Avila from Franz, as well as some of the board members, pulled the conference out of the flames.  We lost money, we lost records, and only last month finally pulled enough of the proceedings together to be able to make a cd out of it.

[...]

The ILC2005 conference went much, much smoother, and we even had proceedings in dead-tree form available _at_ the conference.

[...]

I've never said anything about the ALU to date, because frankly I was always worried for its existence.

With this radical transparency, the success of ILC 2005, Peter Seibel's presence on the board of directors (and his public solication of others' opinions on the Association), this might be a brand new ALU.

Summer Founders

At least two of Y-Combinator's Summer Founders companies are now online.

Reddit is a user-driven popularity contest for URLs.

All of the content on reddit is from users who are rewarded for good submissions (and punished for bad ones) by their peers. You decide what appears on your front page and simultaneously, which submissions rise to fame or fall into obscurity

They seem to be using mod_lisp, TBNL and CMUCL.

At first I thought that a single thumbs-down from me was enough to banish even the #1 link from the front page, but then I figured out that reddit just didn't show me links that I didn't like, and if I logged out I could still see it on the front page with a slightly lower score.

Brain Guppy is basically Am-I-Smart-or-Not, which I have to admit doesn't look like a hit to me. At least with Amihotornot.com you're making superficial evaluations of other people's superficial characteristics. Rapid fire rating of oddly cropped photos of midriff-baring 18-year-olds is its own reward while ranking some nerd's intelligence based on how they answered the question “What's the most surprising thing you've discovered recently?” is at best a chore.

Stuff Guppy seems related.

I think Stuff Guppy and Brain Guppy are both using Ruby on Rails.

Later: Rams points out that ClickFacts, with software to help clients of pay-per-click advertising systems detect click fraud, is another Y-Combinator summer startup.

Even Later: Marc Hedlund says, with inside knowledge, that “At least half of the startups in the program are seriously cool and all of them made a ton of progress on very little money and in very little time. Paul Graham and his partners clearly have good selectors for finding talented, motivated people...” Lucas Carlson, in his post “Y Combinator’s 2005 Summer Founders Program: A Complete Dud”, says that “Some of the sites made during the 24 hours of Rails Day were better than what these guys came up with.”

Most latest: I forgot to link to this comp.lang.lisp post, in which Steve Huffman of Reddit talks a little bit about the Lisp aspects of the site.

My Status

My status is that I am overrun with infection, I am feverish and I am achey. But it's getting better, thanks to antibiotics.

Summer of Lisp Status

Heow Eide-Goodman emailed the Lisp Summer of Code mailing list last week with a status report on the various projects. I've reformatted his message below (the summeroflisp-discuss archives are available only to subscribers, and actually I'm a subscriber and I still can't seem to log in).

Is there a specific end date for the Summer of code?

Update: Luís Oliveira says the deadline is September 1. And Planet SoC aggregates a few Summer of Code blogs (there are many more project-specific SoC aggregators).

Gene-Ontology Database

Is "practically complete"! Being hosted on Common-Lisp.net, Marco and Samantha are now creating the documentation [http://bioinformatics.nyu.edu/~marcoxa/work/cl-godb/index.html]

Fetter-FFI and HelloC-FFI

Rayiner and Luis are working hard and Luis has been making especially good progress.

Axiom UI

From the mouth of Kai:

I wrote some Lisp code (using cl-ppcre and cl-yacc) to convert some LaTeX-like hypertext language to XHTML. I also started writing an Emacs mode for interaction with the new Axiom GUI code. Eventually it will support plots and TeX-rendering of formulas, but right now it doesn't do anything beyond offering basic access to the Axiom GUI back-end.

I will now flesh out the GUI back-end, write some Javascript to connect Firefox to the back-end and change Axiom itself to send all output to the back-end, possibly with some markup to make parsing easier. Once that works (in two weeks maybe) I'll start making it all beautiful and user-friendly.

Tim is happy with Kai's progress and work.

Plt Stepper

John Clements does not have much to show as he's currently between houses. From what he's mentioned, he'll probably end up modifying the UI quite a bit and he expects to have something to show in the next iteration.

ErLisp

Eric Lavigne has been tearing up the ErLisp mailing list and working closely with it's creator Dirk Gerrits and mentor Faré.

They're sort of at a crossroads in regards to choosing which core features to add such as process-linking and error-handling, or work on issues that are most interesting to potential new Erlisp users.

Lots of good stuff going on and this is a good problem to have. :-)

Lisp Sockets

William Bendick isn't going as quickly as he'd like, but should have a library good enough to make public and have people tear apart within a week or so.

Slime Stepper

I don't think I heard from Svein Ove Aas this week, but last report was that his stepper interface that works with SBCL has been completed and a patch submitted. Working on CLisp and MCL next.

PLT DB

Corey Sweeney wrote a checker for his first proposal for the connection interface. and started porting spgsql to my new proposal so people will be able to see it in action when they comment. Right now the connection part works fine, next comes the access part.

However he fears that the spgsql driver needs to be rewritten to make it portable to other schemes.

- Heow

Street Scene 2005

The glamour of rock and roll

The first notable experience of Street Scene 2005 was giving up on the Black Eyed Peas after waiting 45 minutes past their scheduled start time. We ditched that stage right after we saw, on the big video screen, a thrown water bottle graze the head of the poor emcee who came out and tried to calm down a crowd which was growing rapidly more frustrated and angry.

Now that I think about it, maybe the first Official Street Scene Experience was hearing the unmistakeable sound of puke hitting the floor behind me as we rode in a trolley filled with drunk teenagers to Qualcomm Stadium. I didn't turn around, but a friend sitting opposite me said there were definitely potatoes.

Actually, the trolley experience has a tight thematic link to the Black Eyed Peas experience: I heard later that when the Black Eyed Peas finally arrived and complained about being stuck in traffic it appeared that they were completely wasted, a theory strongly supported by the fact that Fergie at some point in the show pissed her pants. (Disturbingly, google searches on this topic reveal that this is not her first Performance Piss.)

But hey, get 75,000 people together for rock and roll and obviously there's going to be some pissing and some puking, if not worse. Luckily things got a lot better after that.

Lori even has some pictures.

Garbage is a well oiled pop machine at this point, and Shirley Manson is going to be some kid's supercool aunt some day (assuming she's not already).

The Killers' songs are really catchy and energetic, and it's fun to see the crowd explode with enthusiasm, but I can't help but agree with the LA Times that the White Stripes are “an infinitely more important band,” and I think I should have bailed on the last few Killers songs to see Jack and Meg. I'm almost surprised to find that I am still a big, unapologetic White Stripes fan.

The Pixies sounded great. But I feel there's something creepily undead about them, that they're perfectly reproducing their old sound but that they lack the life force to create something new. If they sucked the blood out of Fergie it might be a win-win.

I saw a little bit of Snoop Dog. Good, but I think my appreciation was hampered by just having had the Flaming Lips blow the lid off my fucking mind. I hadn't seen them before, and they were smarter and more fun than I remembered rock and roll could be. Lead singer in a hamster ball rolling across the top of the crowd, stage filled with people in furry outfits, kid with light saber, microphone cam. I just kept screaming. They did two beautiful covers: Bohemian Rhapsody to start, and then they closed with War Pigs(!). Best show I've seen in a while, and it turned me into a big fan.

If you know my pal Cody, he was standing next to me, and it was clear that the music contained a coded message keyed specifically for his brain. I think it said “SET INSANITY DIAL TO 11. WAIT--TRY 12.”

Phew.

Man.

Anyway, the show was a blast. And the rest of the weekend, hanging out with Lori, was awesome to the third power^*.

^* Lori found a LiveJournal post that perfectly embodies a facet of the Street Scene experience, I think (that Frank Brown sure did rock).

Baringer's Slime Video

I haven't gotten around to watching Marco Baringer's Slime video (an hour spent watching someone's emacs window is a long time), but I did enjoy his short demonstration of Uncommon Web. I had never seen Uncommon Web-based code before and I felt like the video gave me a good sense of the flavor of writing apps in UCW.

A few notes:

• It was pretty slick when Marco fixed an error in some code while Firefox was blocked trying to display the page, then was able to compile the fixed code and finish serving the request to the browser.
• The HTML that UCW generates looks a little weird, with line breaks inside tags.
• I think I like the gimmick of creating a package named “<”; The macro that generates an HTML paragraph tag is named <:p
• I was impressed that even a trivial form-based app could be created in about 25 lines of code (though I've already heard Ruby on Rails users complain that UCW is too verbose).

It might have been nice if Marco had been able to explain the workings of UCW “actions” in a little more details, but I guess I'll wait for the hour long director's cut (or a post by Bill Clementson) for that.

Grand Challenge Route Closure

DARPA has prohibited teams participating in the 2005 Grand Challenge from performing test runs over the 2004 route. From the addendum to the rules:

The Bureau of Land Management has registered a number of complaints with DARPA regarding the damage Grand Challenge teams have done to the environment in this area. If these activities continue, DARPA risks the loss of its permit to conduct the 2005 Grand Challenge.

Apparently someone was concerned about the turtles after all.