connect-to-inet-socket

dan_b from #lisp suggested that I try to work around the cmucl threading issues by using sockets in non-blocking mode. It was a great suggestion.

I have a new, non-blocking connect-to-inet-socket.

It puts the socket into non-blocking mode, tries to connect, uses system:wait-until-fd-usable (does anyone know what the difference between that and multiprocessing:process-wait-until-fd-usable is?) to nicely block just the current lisp thread until the connect finishes. If the connect worked then the socket is put back into blocking mode and returned.

I use a technique I saw at http://cr.yp.to/docs/connect.html: To figure out whether the async connect succeeded, I call unix:get-peername, and if it returns a negative integer the connect failed. If the connect did fail, in order to figure out the reason for the failure I try to read a single character from the socket with unix:unix-read, which will definitely fail, but more importantly it will set errno according to the reason for the connect failure.

It's been running for a couple days and seems to be working great.

OpenMCL on OS X

OpenMCL 0.10 has been released. It now runs under OS X.

CMUCL Threading

We are running into some of the limitations of cmucl's threading implementation. It's annoying, but mostly because I should have looked this up when we first started using cmucl instead of just checking that multiprocessing was mentioned somewhere in the EncyCMUCLopedia.

It turns out that cmucl has user-level, cooperative threading. Which means that when, for example, I create a Lisp thread and inside it call a function to make a socket connection to some remote host that isn't online, Lisp is completely dead to the world until the connect system call decides to return.

No other Lisp threads will run until that happens. No http requests will be served, no nothing. The listener is dead. (Sometimes so dead that SIGINT isn't enough to wake it up.)

I'm not sure what the best solution to this problem is.

• Try to eliminate all of these sorts of delays; Set socket options, etc. Far from foolproof, lots of opportunity to miss something, or to have a delay inside code that is not under your control.
• Try ACL under Linux. It uses user-level threading under unix, but maybe it is preemptive. Oops, I just checked and foreign code, at least, will not be preempted. Never mind.
• Actually do some forking and run anything that could take a significant amount of time in a separate process.
• Check whether Douglas T. Crosher has finished his kernel-level threads package for cmucl, and how much it costs.
• Turn on cmucl's SIGALRM thread preemption mechanism. This feature is never mentioned by anyone without saying in the same breath "cmucl's code is not guaranteed to be thread-safe!"

Like I said, annoying.

Why Python For AIMA

Peter Norvig's rationale for why the next edition of AI: A Modern Approach will include Python code as well as Lisp code:

1. Python is easier to support cross-platform; there is one standard, free Python implementation that runs on Windows, MacOS, Linux, and most other Unix versions. With Lisp, you need to support several different implementations to hit all the platforms. This has proven to be a (minor) maintainence problem.
2. Python has standard libraries for http, html, xml, regexs, database integration, persistent storage, interfacing to C, threads, etc. Lisp has most of these things in particular implementations, but they are not standard. (The Lisp standard was a marvel in 1992, but it froze then, and other languages have leaped ahead.)
3. Python has a cross-platform GUI toolkit; Lisp does not.
4. Of all the established languages in the world, Python is closest to the pseudo-code that we developed (independently) for the book.
5. Python has an active, visible community. The Lisp community is active too, but not as visible: in the last month there were 28,000 news posts to *.python.*, and 10,000 to *.lisp.*. Partly this is because the Lispers already know things and don't need to ask, while Python is aimed more at beginning programmers, but these numbers do indicate some measure of interest.
6. Some people are parenphobic and just don't like Lisp. For them, and for the sake of variety in general, it's good to have a choice.
7. See my [Norvig's] Lisp vs. Python comparison for more rationale on using Python.

Most of these I can at least sympathize with. The lack of standard libraries is definitely a pain. And while most of the cross-implementation issues are relatively minor, they exist. (Don't tell me that everything will be OK if you just write ANSI conforming code. I try to, and it just brings into relief the fact that no Lisp implementation is perfectly conforming, and that conformance bug fixes can take a long time to be released.)

The rest I think are more relevant to a textbook author than to an application developer. Maybe I am sheltered, but does anyone really use cross-platform GUI kits for commercial code?

FFI debugging

Echo under cmucl was hanging; the web server wasn't responding and the listener was mostly dead. Ctrl-c

MCL has some "dcmds;" that allow you to get some information from the running lisp image once you've broken into the lowlevel Mac debugger macsbug. The plbt command displays a stack backtrace that includes both Lisp functions and foreign functions:

MCL on OS X: Coming Soon

Well, I installed OS X, but MCL 4.3.1, which is the latest available, only runs in Classic mode. The good news appeared on the info-mcl mailing list a couple weeks ago, when Alexander Repenning posted a link to a screenshot of a beta version of MCL running native:

ACL 6.1 Patched

Franz patched the emacs-lisp interface slowness that was keeping me from upgrading to 6.1:

  Thu Jan 17 12:06:47 PST 2002
   Patch: acl612.dll, acli612.dll
    Windows only: Corrects a problem on windows where winsock was not
    waking a thread waiting on i/o, sometimes causing 1+ second long
    delays when activating a thread. This was particularly noticeable
    when using ACL via the Emacs-Lisp interface.
   Impact: Recommended for ELI users on Windows platforms.

Mike installed it, and it works. Lisp in emacs is fast again.

3D Development

While playing around with MCL the other day, I revisited the QuickDraw 3D interface I wrote a couple years back. In a few hundred lines of code I taught MCL's inspector to understand QD3D objects:

? (q3:qd3d-inspect (q3:read-3dmf-file "tree.3dmf"))

The QD3D inspector allows you to select and inspect components of an object, highlight mesh faces, and delete mesh faces.

I'm sure everybody on the Mac has made the move to OpenGL from QuickDraw 3D, so it's not worth putting this into a new release of the QD3D interface. This exercise did remind me that the MCL development environment is still the best of all the lisps, by far. And it reminded me of what a clean API QD3D was. They were a total blast to use together, interactively, to explore 3D graphics programming.

In the next week I think I may install OS X, and play with MCL some more.

The Tendency of Unresolved Bugs

For only the second time ever (well, we've only been keeping track for a month), the number of unresolved bugs has gone down:

Inside Orbitz

An interesting but short description of ITA Software's very real-world lisp application: Inside Orbitz.

  4. Because we have about 2 gigs of static data we need rapid access
  to, we use C++ code to memory-map huge files containing pointerless
  C structs (of flights, fares, etc), and then access these from
  Common Lisp using foreign data accesses. A struct field access
  compiles into two or three instructions, so there's not really any
  performance penalty for accessing C rather than Lisp objects. By
  doing this, we keep the Lisp garbage collector from seeing the data
  (to Lisp, each pointer to a C object is just a fixnum, though we do
  often temporarily wrap these pointers in Lisp objects to improve
  debuggability). Our Lisp images are therefore only about 250 megs of
  "working" data structures and code.

I like this peek into the low-level implementation guts of industrial strength lisp code. They have about 10 seconds to search 10^30 fares, and they do it by not consing, looking at the disassembly of every lisp function and using foreign code where neccessary. And it sounds like they are winning in the marketplace.

Also see the slashdot discussion and the Franz article.

A note regarding not consing: I am not sure that it is the guaranteed path to high performance so many people think it is; Lisp garbage collectors are pretty good these days and it might actually be faster for you to generate the garbage and let the GC take care of it, compared to writing your own resourcing code. It's also true that the GC has fewer bugs in it right now than the first version of any resourcing code you are going to write. This post by David Lamkins touches on the issues involved. Anyway, I wouldn't automatically assume resourcing was going to win me any performance in a given situation.

On Hacker

From a comp.lang.lisp article by Kent Pitman on the Jargon file's "Hacker" entry:

  And, to my knowledge, it just about NEVER referred to anything related
  to UNIX.  That's historical revisionism on Raymond's part, IMO,
  because he was too afraid to name the operating systems of the time
  and instead wanted to say something anachronistic that would give a
  modern reader the sense of what was revered back then.

I like that word. revered.

Alex Moffat and Ehud Lamm

I like Alex Moffat's I Thought You Might Be Interested weblog, even though it focuses mostly on java. He just sounds like a thoughtful programmer.

Ehud Lamm's Lambda The Ultimate, on the other hand, looks like it would be cool if you were into programming language design and research, which I am not. I can't be the only one that mostly doesn't care about a new little language that one guy designed and implemented and gets used by a dozen people. Still, there might be something interesting there.

I Must Soon Quit The Scene

If this page had a soundtrack it would be i must soon quit the scene / american analog set

CMUCL Bugs

I sent in a little patch to cmucl.

Mike suspected that cmucl's buggy connect-to-inet-socket function was the one responsible for the file descriptor leak we were seeing in Echo, the one that locked it up after a day or so when it had 1024 open files and the operating system put a stop to things. He was right.

connect-to-inet-addr wouldn't connect to an address given in ipaddr form if it couldn't lookup the hostname corresponding to that IP, and it leaked a file descriptor every time it errored out in that way (it would also leak a descriptor if you gave it a hostname that it couldn't resolve).

There was reluctance by some cmucl developers to believe that it was a bug to be unable to connect to an IP given in ipaddr format for which a reverse DNS lookup could not be done. I don't know why.

So I fixed it.

It would have been somewhat easier to track down what was going wrong if the linux proc filesystem had some more documentation. Like, how do I find out what this is:

lrwx