dan_b from #lisp suggested that I try to work around the cmucl threading issues by using sockets in non-blocking mode. It was a great suggestion.
I have a new, non-blocking connect-to-inet-socket.
It puts the socket into non-blocking mode, tries to connect, uses system:wait-until-fd-usable (does anyone know what the difference between that and multiprocessing:process-wait-until-fd-usable is?) to nicely block just the current lisp thread until the connect finishes. If the connect worked then the socket is put back into blocking mode and returned.
I use a technique I saw at http://cr.yp.to/docs/connect.html: To figure out whether the async connect succeeded, I call unix:get-peername, and if it returns a negative integer the connect failed. If the connect did fail, in order to figure out the reason for the failure I try to read a single character from the socket with unix:unix-read, which will definitely fail, but more importantly it will set errno according to the reason for the connect failure.
It's been running for a couple days and seems to be working great.
OpenMCL 0.10 has been released. It now runs under OS X.
We are running into some of the limitations of cmucl's threading implementation. It's annoying, but mostly because I should have looked this up when we first started using cmucl instead of just checking that multiprocessing was mentioned somewhere in the EncyCMUCLopedia.
It turns out that cmucl has user-level, cooperative threading. Which means that when, for example, I create a Lisp thread and inside it call a function to make a socket connection to some remote host that isn't online, Lisp is completely dead to the world until the connect system call decides to return.
No other Lisp threads will run until that happens. No http requests will be served, no nothing. The listener is dead. (Sometimes so dead that SIGINT isn't enough to wake it up.)
I'm not sure what the best solution to this problem is.
Like I said, annoying.
Most of these I can at least sympathize with. The lack of standard libraries is definitely a pain. And while most of the cross-implementation issues are relatively minor, they exist. (Don't tell me that everything will be OK if you just write ANSI conforming code. I try to, and it just brings into relief the fact that no Lisp implementation is perfectly conforming, and that conformance bug fixes can take a long time to be released.)
The rest I think are more relevant to a textbook author than to an application developer. Maybe I am sheltered, but does anyone really use cross-platform GUI kits for commercial code?
Echo under cmucl was hanging; the web server wasn't responding and the listener was mostly dead. Ctrl-c
MCL has some "dcmds;" that allow you to get some information from the running lisp image once you've broken into the lowlevel Mac debugger macsbug. The plbt command displays a stack backtrace that includes both Lisp functions and foreign functions:
Well, I installed OS X, but MCL 4.3.1, which is the latest available, only runs in Classic mode. The good news appeared on the info-mcl mailing list a couple weeks ago, when Alexander Repenning posted a link to a screenshot of a beta version of MCL running native:
Franz patched the emacs-lisp interface slowness that was keeping me from upgrading to 6.1:
Thu Jan 17 12:06:47 PST 2002 Patch: acl612.dll, acli612.dll Windows only: Corrects a problem on windows where winsock was not waking a thread waiting on i/o, sometimes causing 1+ second long delays when activating a thread. This was particularly noticeable when using ACL via the Emacs-Lisp interface. Impact: Recommended for ELI users on Windows platforms.
Mike installed it, and it works. Lisp in emacs is fast again.
? (q3:qd3d-inspect (q3:read-3dmf-file "tree.3dmf"))
The QD3D inspector allows you to select and inspect components of an object, highlight mesh faces, and delete mesh faces.
I'm sure everybody on the Mac has made the move to OpenGL from QuickDraw 3D, so it's not worth putting this into a new release of the QD3D interface. This exercise did remind me that the MCL development environment is still the best of all the lisps, by far. And it reminded me of what a clean API QD3D was. They were a total blast to use together, interactively, to explore 3D graphics programming.
In the next week I think I may install OS X, and play with MCL some more.
For only the second time ever (well, we've only been keeping track for a month), the number of unresolved bugs has gone down:
4. Because we have about 2 gigs of static data we need rapid access to, we use C++ code to memory-map huge files containing pointerless C structs (of flights, fares, etc), and then access these from Common Lisp using foreign data accesses. A struct field access compiles into two or three instructions, so there's not really any performance penalty for accessing C rather than Lisp objects. By doing this, we keep the Lisp garbage collector from seeing the data (to Lisp, each pointer to a C object is just a fixnum, though we do often temporarily wrap these pointers in Lisp objects to improve debuggability). Our Lisp images are therefore only about 250 megs of "working" data structures and code.
I like this peek into the low-level implementation guts of industrial strength lisp code. They have about 10 seconds to search 10^30 fares, and they do it by not consing, looking at the disassembly of every lisp function and using foreign code where neccessary. And it sounds like they are winning in the marketplace.
A note regarding not consing: I am not sure that it is the guaranteed path to high performance so many people think it is; Lisp garbage collectors are pretty good these days and it might actually be faster for you to generate the garbage and let the GC take care of it, compared to writing your own resourcing code. It's also true that the GC has fewer bugs in it right now than the first version of any resourcing code you are going to write. This post by David Lamkins touches on the issues involved. Anyway, I wouldn't automatically assume resourcing was going to win me any performance in a given situation.
And, to my knowledge, it just about NEVER referred to anything related to UNIX. That's historical revisionism on Raymond's part, IMO, because he was too afraid to name the operating systems of the time and instead wanted to say something anachronistic that would give a modern reader the sense of what was revered back then.
I like that word. revered.
I like Alex Moffat's I Thought You Might Be Interested weblog, even though it focuses mostly on java. He just sounds like a thoughtful programmer.
Ehud Lamm's Lambda The Ultimate, on the other hand, looks like it would be cool if you were into programming language design and research, which I am not. I can't be the only one that mostly doesn't care about a new little language that one guy designed and implemented and gets used by a dozen people. Still, there might be something interesting there.
I sent in a little patch to cmucl.
Mike suspected that cmucl's buggy connect-to-inet-socket function was the one responsible for the file descriptor leak we were seeing in Echo, the one that locked it up after a day or so when it had 1024 open files and the operating system put a stop to things. He was right.
connect-to-inet-addr wouldn't connect to an address given in ipaddr form if it couldn't lookup the hostname corresponding to that IP, and it leaked a file descriptor every time it errored out in that way (it would also leak a descriptor if you gave it a hostname that it couldn't resolve).
There was reluctance by some cmucl developers to believe that it was a bug to be unable to connect to an IP given in ipaddr format for which a reverse DNS lookup could not be done. I don't know why.
So I fixed it.
It would have been somewhat easier to track down what was going wrong if the linux proc filesystem had some more documentation. Like, how do I find out what this is: