It’s The End Of The World As We Know It, And I Smell Poo

Doug Harvey wrote a very nice account of the CLUI tour to Terminal Island.

Gumaer directs the bus driver through the enormous system of pipes, centrifuges and settling tanks before coming to a halt by the aeration basins, where we disembark for a closer look at the bubbling microbial soup, with its eddies of gold disposable tampon applicators and hovering cloud of septic microbe farts. I overhear one of the other tourists laugh, “Only Matt could get people to pay to get up at 7 a.m. to stand and stare at tubs of shit!”

MCL 5.1 Shipping

MCL 5.1 is shipping! There are lots of changes and fixes, including more MOP support and what looks like better event handling in the OS X model.

I was thinking recently that Digitool might be down for the count, given the extremely low level of visible activity, but this at least bodes well.

Simbob

I think this is kind of funny:

>>> import simbob2
Dispatching sim event of type 'image', timestamp 1106039342.1
INFO  - Ignoring motion-triggered image that is too old (11105082.168618 s old).
Dispatching sim event of type 'image', timestamp 1106040189.54
INFO  - Ignoring motion-triggered image that is too old (11104234.818464 s old).

The simulation's ties to the real world are not yet completely cut. Most of its perception is synthetic, playing from a tape recorded 128 days ago, on Jan 18, 2005 at 8:18 in the morning. The simulation's only remaining connection to the real world, its idea of what the current time is, is a bug that will be fixed in the next revision and then it will be cut entirely free from reality as it happens right now.

It's like TRON meets a Philip K. Dickian mindfuck.

Switcher

Gavin tipped me off a while ago that Joel Reymont, the guy with the unusual background who was writing an industrial strength poker server in Lisp, switched to Erlang.

It took him 6 weeks to rewrite his server, which can now handle 27,000 simultaneous games on his Powerbook.

Joel first looks at Erlang:

I got depressed looking at Erlang today.

Joel decides to switch languages:

I want to scale well above 10,000 users. I want load balancing, fault tolerance, failover, a database that lets me store objects without writing a lot of code, etc. With all the bells and whistles that Allegro CL provides it does not help me with the above and Erlang does. And it's free too.

Joel talks more about why he switched to Erlang:

You can build a server that handles a few hundred users in any programming language. You cannot compete with such a server, though. You need a design that lets you cluster together a few poker servers, adding more as needed. You need a live poker server to take over when another one dies. This has to work seamlessly and be easy to setup and maintain. This is a prerequisite, the membership to the club if you will. You need to offer more to compete with the big guys and this is where the going gets tough. Think zero configuration, 100% uptime and unlimited scalability.

Allegro CL is known for coming with everything and the kitchen sink. Web server, web services, RPC, you name it and it's included. The price bites but the customer service is great. There's nothing in Allegro CL that helps me compete with the big boys, though. I have to build it myself

[...]

Erlang was designed to handle problems like this. Fail-over, take-over and load balancing are built-in concepts

I'm a little disappointed because I thought his project was going to make a great Lisp success story, but it sounds like he made the right choice for his application, and for his business. I'm curious to know how it turns out.

Image Abuse

I love it when I wake up in the morning and my RSS feeds are clogged with some fun, clever thing a friend of mine has done.

A couple years ago I was at the MJT for a rare quadruple feature consisting of Road House, Corvette Summer, Glitter and At First Sight (there was a lot of fast forwarding). Some guy sitting behind me kept making funny, insanely clever comments throughout the night. It turned out to be Nat. Besides hacking Amazon URLs and being generally very busy, Nat also happens to be editing a collection of comics from 24 Hour Comics Day.

E3: More, More, More

From Lore Sjöberg's wrap up of E3:

This time around, we have the same players as the last generation, and nothing giving a clear technological advantage to any one console. Instead of a battle of technology, the next couple of years are shaping up to be a battle of marketing, a much less exciting prospect.

CLiki Grease

Photo by Frank Kolodziej

Since Greasemonkey is so cool and is revolutionizing the web (get in line), I figured I should play with it a little bit. Anything to help me when the Dashboard, Ajax and Greasemonkey people all gang up and force us into Javascript reeducation camps.

sane-cliki.user.js attempts to fix CLiki's handling of HTML entities when editing pages (see my previous rant). Due to the ways browsers try to correct for mangled HTML in <TEXTAREA> tags, it wasn't possible to just grab the source via DOM and encode it properly; I had to make an additional request to the CLiki server to get the unadulterated page source. This does mean that every time you hit the “Edit” link at the bottom of a CLiki page, two requests will be sent to the CLiki server and you have to wait for both of them to complete. And yes, it means this single script uses both Greasemonkey and XMLHttpRequest, which is like a double hip web takedown.

A page as it appears rendered by CLiki.

Left: Mangled HTML in the unfixed CLiki. Right: Fixed courtesy of Greasemonkey.

This is a hacky way of fixing the problem, but it works until CLiki is updated. And when I say it “works,” I mean that I tried it at least twice and it crashed at most once. The script requires, of course, Firefox and Greasemonkey. One could probably do something similar with PithHelmet for Safari.

Mark Pilgrim's Dive Into Greasemonkey was very helpful.

ECLM Audio

Fire Escape, Sarah Sze, 2002

Over a week without DSL left me with a little catching up to do.

Kevin Smith read my whining about the European Common Lisp Meeting videos, and my call for someone to extract just the audio, which I hoped might deliver more bang per gigabyte downloaded, and decided to do something about it. In both mp3 and ogg format.

Terminal Island

This weekend Lori and I went on a CLUI bus tour of Terminal Island, part of their exhibit on the the island, which is the home of the Port of Los Angeles.

I took some crappy pictures with my cell phone.

The tour was excellent, as I am becoming more confident that all CLUI tours are. Matt from the Center knows his stuff inside and out, and is always able to use the statistics and figures he has internalized to give insights into the larger picture. It's a great way to understand the enormous trade deficit in a concrete sense.

The only disappointment was that tickets sold out so fast that JoAnne, who I think of as our CLUI bus tour partner, didn't have a chance.

Comic I

This is the first comic I did for 24 Hour Comics Day.

ECLM Audio

Someone should extract the audio from the European Common Lisp Meeting videos and make that available for download in an attempt to salvage something worthwhile from the effort. There's no way I'm going to download 4 gigs of video, shot from the very back of the room, with a tiny little presenter visible off in the distance.

I've seen videos shot secretly by people fearing prosecution and confiscation of their equipment if they were discovered that looked better. Since I can't see what the presenter is doing, just give me the straight up audio, which would be a 200 MB download at most.

It's too bad, I would have loved to have seen what was going on in the SLIME talks.

Update: Markus Fix has the audio from Luke Gorrie's SLIME talk. Yay! Though it does seem possible that in most of the videos it's not possible to either see or hear what's happening. Oh well.

Radar Bee Love

A bee wearing the latest in radar transponder couture.

“Waggle dance controversy resolved by radar records of bee flight paths” [via robotwisdom].

Exceptions

The other day in #lisp, there was some discussion of the trivial-http library and situations in which it might leak file descriptors.

Here's some of the relevant code from trivial-http:

(defun http-get (url)
  (let* ((host (url-host url))
         (port (url-port url))
         (stream (open-stream host port)))
    (format stream "GET ~A HTTP/1.0~AHost: ~A~AUser-Agent: Trivial HTTP for Common Lisp~A~A"
        url +crlf+ host +crlf+ +crlf+ +crlf+)
    (force-output stream)
    (list
     (response-read-code stream)
     (response-read-headers stream)
     stream)))

I'm embarrassed to say that at first I didn't get it; I didn't see how anything could be leaked, and the interface looked so simple, how could one consider improving it?

Then, coincidentally, the next day I read Joel Spolsky's “Making Wrong Code Look Wrong” article. At the very end he says he doesn't like exceptions. Oh god, I thought, isn't it enough that I have to write tons of error-checking C++ code at work because we don't use exceptions? Now Joel Spolsky is telling everyone not to use them?

Here’s the thing with exceptions, in the context of this article. Your eyes learn to see wrong things, as long as there is something to see, and this prevents bugs. In order to make code really, really robust, when you code-review it, you need to have coding conventions that allow collocation. In other words, the more information about what code is doing is located right in front of your eyes, the better a job you’ll do at finding the mistakes. When you have code that says

dosomething();
cleanup();

… your eyes tell you, what’s wrong with that? We always clean up! But the possibility that dosomething might throw an exception means that cleanupmight not get called. And that’s easily fixable, using finally or whatnot, but that’s not my point: my point is that the only way to know that cleanup is definitely called is to investigate the entire call tree of dosomething to see if there’s anything in there, anywhere, which can throw an exception, and that’s ok, and there are things like checked exceptions to make it less painful, but the real point is that exceptions eliminate collocation. You have to look somewhere else to answer a question of whether code is doing the right thing, so you’re not able to take advantage of your eye’s built-in ability to learn to see wrong code, because there’s nothing to see.

OK, I guess the part about inspecting the entire call tree is true, but the way to save yourself from that tedium is to just always assume that an exception could be thrown (especially since you have to consider the fact that the code you're calling may change when you upgrade libraries, etc.--whoa, something that checked exceptions actually help with). Not a huge deal. But what this reminded me of was Richard Gabriel's and Paul Graham's complaints about object oriented code, which I agree with. Richard Gabriel describes code inheritance as “compression”, where to understand the meaning of any particular (in Lisp terms) generic function call you need to understand the meaning of every method on that function defined by superclasses (and in Lisp, subclasses). Paul Graham says that “Object-oriented programming offers a sustainable way to write spaghetti code.”

So Spolsky has the same complaint with exceptions that Gabriel and Graham have with object orientation, and now I realize that use of exceptions does indeed have a cost. I think I'll probably keep using them anyway, just as I often find object oriented code useful. Or, as Raymond Chen puts it, exceptions are “cleaner, more elegant and harder to recognize”, because he argues that evaluating the correctness of exception-based code is harder than evaluating the correctness of error-code-based code. “But that's okay. Like I said, just because something is hard doesn't mean it shouldn't be done. It's hard to write a device driver, but people do it, and that's a good thing.”

Once I was primed, it was easy to see the problem with trivial-http:http-get: if response-read-code or response-read-headers or format or any other line of code in the body of the let fails, the stream will never get closed. And please, do not try to argue that the garbage collector will close the stream (that's not how Lisp works), or should be made to do it using explicit finalizers/termination (it's unclean and nonportable).

So now I understand the problem (well, maybe--I still wonder how you would change the interface to avoid it without coming up with an ugly interface). What's the best way to fix it? The following code isn't extremely pretty:

(defun http-get (url)
  (let* ((host (url-host url))
         (port (url-port url))
         (stream (open-stream host port))
         (success-p NIL))
    (unwind-protect
        (progn
          (format stream "GET ~A HTTP/1.0~AHost: ~A~AUser-Agent: Trivial HTTP for Common Lisp~A~A"
                  url +crlf+ host +crlf+ +crlf+ +crlf+)
          (force-output stream)
          (let ((result (list (response-read-code stream)
                              (response-read-headers stream)
                              stream)))
            (setf success-p T)
            result))
      (unless success-p
        (close stream)))))

Can we do better than that? I'm not sure. We can at least macroize this idiom:

(defmacro with-non-local-cleanup (protected-form cleanup-form)
  "Like unwind-protect, but only runs the cleanup-form in the
   case of a non-local exit, not if protected-form terminates
   normally."
  (let ((success-var (gensym)))
    `(let ((,success-var NIL))
       (unwind-protect
            (multiple-value-prog1
                ,protected-form
              (setf ,success-var T))
         (unless ,success-var
           ,cleanup-form)))))

(defun http-get (url)
  (let* ((host (url-host url))
         (port (url-port url))
         (stream (open-stream host port)))
    (with-non-local-cleanup
        (progn
          (format stream "GET ~A HTTP/1.0~AHost: ~A~AUser-Agent: Trivial HTTP for Common Lisp~A~A"
                  url +crlf+ host +crlf+ +crlf+ +crlf+)
          (force-output stream)
          (list
           (response-read-code stream)
           (response-read-headers stream)
           stream))
      (close stream))))

All this leaking reminds me of an old bug in CMUCL's connect-to-inet-socket function.

;; This code is from 2002.
(defun connect-to-inet-socket (host port &optional (kind :stream))
  "The host may be an address string or an IP address in host order."
  (let ((socket (create-inet-socket kind))
        (hostent (or (lookup-host-entry host)
                     (error "Unknown host: ~S." host))))
    (with-alien ((sockaddr inet-sockaddr))
      (setf (slot sockaddr 'family) af-inet)
      (setf (slot sockaddr 'port) (htons port))
      (setf (slot sockaddr 'addr) (htonl (host-entry-addr hostent)))
      (when (minusp (unix:unix-connect socket
                                       (alien-sap sockaddr)
                                       (alien-size inet-sockaddr :bytes)))
        (unix:unix-close socket)
        (error "Error connecting socket to [~A:~A]: ~A"
               (host-entry-name hostent)
               port
               (unix:get-unix-error-msg)))
      socket)))

I find it a little funny that the part of that function that works with unix error codes correctly closes the socket when something goes wrong, while the code that uses more modern concepts of error handling (the (error "Unknown host...) bit) leaks a file descriptor when there's a problem (I also still find it funny that CMUCL wouldn't allow you to open a socket unless you could do a reverse DNS lookup of the target address). Maybe this is evidence for Raymond Chen's assertions, and whoever wrote this code was conscientious enough to feel weird about calling unix:unix-connect without checking the return value, but missed the problems with exceptions, even exceptions that the programmer explicitly raised himself.

PM

I'm a little freaked out about ultrafine particulate matter (PM) pollution. It seems like I've been reading more about it lately, and the increased risk of death (including “sudden death”) associated with it. What's scary is despite the fact that there's a clear link between premature death and the levels of particulate pollution of a given city (with 50,000 deaths each year attributed to PM in the United States, and 500,000 deaths per year worldwide) the mechanisms involved aren't very well understood.

The particles in the photos above (i) are less than 0.1 μm in diameter, and hence unregulated by the government, (ii) are the major component in vehicle emissions and (iii) have gotten inside the mitochondria of the cells that are supposed to clean up foreign crap and are killing them. Additionally, my state of mind is not helped by the fact that these particular particles were collected in the LA basin--you know, where I live--and that this weekend's move has made even more obvious the incredible amounts of grime that float in from Sunset Blvd. and coat every surface of my apartment. Including, you know, my lungs.

Actually, lungs may not get the worst of it. It looks like PM can damage the nasal passages' ability to block foreign substances from entering the brain. In a study of 200 dogs living in Mexico City, investigators report “tracing metals associated with fossil fuel combustion—chiefly vanadium and nickel—from the dogs' nasal tissue, through the olfactory bulb, and into the frontal lobe and hippocampus of the animals' brains.” Aieee.

Because such metals can foster damage by generating free radicals, Calderón-Garcidueñas looked for signs of brain changes in dogs living in areas with heavy particulate pollution.

Dogs often serve as a model for human age-related cognitive impairments. Some dogs at age 10 and older develop the waxy brain plaques characteristic of Alzheimer's disease. “In Mexico City,” Calderón-Garcidueñas told Science News, “we are seeing [plaque] pathology in 11-month-old pups”—a dramatic acceleration in the development of the signature of Alzheimer's disease.

These data are “definitely worrisome,” she says, especially in light of her preliminary findings of a similar breakdown in the nasal tissue of many people living in Mexico City.

LA has the worst 2.5 μm PM pollution in the country. And you're not going to find an air filter at Home Depot that can deal with that, let alone 0.1 μm particles.

Nerdtasia

Everything is chaotic here at lemonodor right now. My work life is a little unsettled, I was out of town for a few days, and Lori and I just moved to a larger apartment. Hence a recent paucity of lemon posts. Maybe I'll name the units of posts here “licks,” as in “lemon licks”. You have not received any lemon licks in a while, and for that I am sorry.

I went on a little bay area jaunt, which was kind of whirlwindy. The main objective was to see Paul Graham's talk at PARC. Paul's entertaining, and he speaks from an authority that in part comes from having made terrible, embarrassing mistakes and not being afraid to tell the world about them. Unfortunately I think I'm still stuck on the part of starting a company that involves coming up with a good idea, so the subject of his talk, selling your startup, is a pretty theoretical concept for me.

I barely chatted with Paul, exchanged a few words with the always-smiling Trevor Blackwell, was groped amateurishly by Aaron Swartz (I kid, I'm a fan of Aaron's!), and got a signed copy of Practical Common Lisp from Peter Seibel. Notably, I digested an entire La Costena burrito during the talk, courtesy of Gavin.

After the talk I went to dinner with a bunch of 106 Miles people, where the awesome Adam Rifkin took me under his wing and introduced me to various Technorati, Jotspotters, Dojoers and others who are knee-deep in the Web 2.0.

I also had time for a compressed visit to the Computer History Museum. I expected it to be moldy and arcane and irrelevant, but it totally blew my mind. Or maybe I just like moldy and arcane.

The next day I hung out with dnm and learned a little bit about his secret plans for Lisp-based domination. “My friend and I decided we hated [censored website], and so we decided to do something that would kill [censored website].”.

Later I met with Franz. They showed me some of their latest work (me: “You guys are nuts. In a good way.”) and pumped me for all sorts of information, which I was all too happy to give them after a little flattery. All you need to do is call me a rock star once, it's easy. Actually, everyone at Franz was delightfully open and friendly (I think for some reason this surprised me) and I had a good time.

The next night est and Ruchira made sure I got to go to the Google engineering open house. Whoa, talk about a nerdtastic nerdsplosion. “Powerpoint comedian” Don McMillan totally killed, and it was probably one of a dozen venues on Earth where that's even possible. Luckily I prefetched a pair of margaritas before the presentations even began. I think I met about 150 people through Adam and Joyce, and then stood around while Guido and Peter Norvig and est talked language design. Eric and Ruchira, gracious hosts that they were, made sure I talked to all the cool people. I even ran into Paul Snively, a CRACL-head from LA who I hadn't seen in a while.

The biggest coincidence was finding out that my old grad school advisor's office is 15 feet from est's cubicle. On second thought, I should have expected that.

A lot of people I met knew of lemonodor, which was a nice surprise. Actually, it lent an air of surreality to the whole trip, which was odd enough just because there's no place in LA where I run into more than a half-dozen hardcore geeks at once, while in Palo Alto and Mountain View they are legion. It's a different world.

Paul Graham's Berkeley Talk

Federico Solmi, Rush To Hospital II

Dan Moniz reports from Paul Graham's “Hiring is Obsolete” talk last night at Berkeley, including some Lisp celebrity sightings.

Starting Up

Pinku's original, unedited post on startups described his experience at the company I worked for as my first job out of grad school. I even remember the particular meeting he (originally) mentioned. The edited post has a more optimistic tone.

Yeah, we were trying to commercialize some AI technology, originally implemented in Lisp.

Explorers Club: Playas

Matt kicks off another Explorers Club meeting

Last night at CLUI, Steve Rowell reported on his recent work in Playas, New Mexico, a tiny (260 homes) company town once run by the Phelps Dodge Mining Company. The entire town of Playas was recently purchased for $5 million by the Dept. of Homeland Security so it could be turned into an anti-terrorist training center.

Instead of a ribbon cutting ceremony to mark the occasion of the sale, there was a ribbon detonating ceremony. After which a simulated bomb blew up a bus and simulated bus bombing victims staggered away, carrying simulated babies and holding simulated severed limbs.