The Guitar Player's Girlfriend

Photo by Lori

Last week Lori, Carrie and I were extras in a friend's short film. We played indie rockers watching a band in a bar. Standing in a hazy club with a bunch of hipsters, beer in hand and nodding to some guitar-based rock and/or roll is not exactly a foreign concept, but doing it as simulation was an odd way to recall that I haven't actually done it for real in a while and that I miss it.

This was the first time I've participated first-hand in the stereotypical Los Angeles industry as opposed to just observing. Maybe I should take advantage of the free time I have these days and have some fun as an extra. Next stop on the background actor train, a Hollywood blockbuster [via robotwisdom]:

Before Take 7 Petersen comes on. “One more,” he says. “Now with full acting. Let's make this the one!” It works. As the first AD calls “Action,” it's like the entire background cast has been hit with a Taser. The place goes berserk, fear and chaos cranked to 11. And I'm swept up in it.

Too bad I didn't think of this while they were shooting Miami Vice.

Border Patrol UAV Crash

I missed this until now, but a U.S. Border Patrol's Predator UAV crashed in Arizona last month. The crash didn't make civilian pilots too happy.

“This crash highlights the safety concerns we've voiced all along,” said Andy Cebula, AOPA executive vice president of government affairs. “The FAA should not permit UAV operations until they are certified to the same level of safety as manned aircraft.”

“Thankfully, in this accident no one in the air or on the ground was hurt,“ Cebula added. ”But just think that if a pilot had been flying legally under the TFR and the UAV hit the aircraft from behind and above -- the pilot would have had no chance to see and avoid the uncontrolled UAV.”

The Aircraft Owners and Pilots Association wasn't too happy about the UAVs before they crashed, either, since the FAA had imposed flight restrictions on a 350 mile section of Arizona and New Mexico because of the Predators' operations. And in general the AOPA worries about the danger UAVs pose to other aircraft.

As it turns out, the preliminary NTSB report on the accident seems to indicate that the crash was due to pilot error:

The pilot reported that during the flight the console at PPO-1 "locked up", prompting him to switch control of the UAV to PPO-2. Checklist procedures state that prior to switching operational control between the two consoles, the pilot must match the control positions on the new console to those on the console, which had been controlling the UAV. The pilot stated in an interview that he failed to do this. The result was that the stop/feather control in PPO-2 was in the fuel cutoff position when the switch over from PPO-1 to PPO-2 occurred. As a result, the fuel was cut off to the UAV when control was transferred to PPO-2.

The pilot stated that after the switch to the other console, he noticed the UAV was not maintaining altitude but did not know why. As a result he decided to shut down the GCS so that the UAV would enter its lost link procedure, which called for the UAV to climb to 15,000 feet above mean sea level and to fly a predetermined course until contact could be established. With no engine power, the UAV continued to descend below line-of-site communications and further attempts to re-establish contact with the UAV were not successful

The U.S. Border Patrol isn't the only group flying UAVs in the area. A vigilante group called the American Border Patrol, in conjunction with the Minutemen, has been flying their own homebuilt UAV—apparently a souped up model airplane. They even have video of their vehicle in action.

And then there is the anti-vigilante group UAV flown near the border by Chris Csikszentmihalyi. Quite the crowded airspace down there these days.

Montezuma Heat Death

I decided to spend a little time optimizing Montezuma indexing for speed, with the hope that I could get it to index the Planet Lisp archives within a reasonable amount of time.

Indexing 1000 pastes took about 100 seconds, as I mentioned the other day. Avoiding 6 million garbage-generating calls to subseq saved 10 seconds. Cutting down on the number of calls to string< and string> saved 4 seconds. Each was a significant speedup, but nothing dramatic.

After doing some of this benchmarking, the laptop I was using overheated and shut down. Since this time there were no cats lying on top and therefore no obvious scapegoats, I thought maybe the vents had just gotten clogged and so I blasted the hell out of it with canned air.

The comprehensive air blast cleaning cut 45 seconds off the benchmark time. It was like getting a processor upgrade.

The McClouds' 50 State Tour

To celebrate Scott's new book, Making Comics, the McClouds are going to drive through all 50 states.

In addition to the public appearances, the whole family will be blogging our experiences on the road and interviewing comics artists around the country about their work. We plan to have a lot of fun (when we're not going crazy or trying to kill each other) and we hope you'll be part of it all when we come to your state.

Desperate Petunia

Yesterday I had one cat-induced thermal shutdown of that machine, plus one ctrl-paw-delete.

Montezuma Charges Along

Photo by Peter Menzel

Montezuma is coming along pretty well. Indexing is complete and seems mostly bug free, and Gary King and I are finishing off the support for searches. We've got simple term queries, phrase queries, wildcard queries, range queries and boolean queries pretty much working.

For testing, I grabbed 1000 pastes from the paste service at paste.lisp.org. Each paste consists of a number (the unique paste ID), the user who made the paste, the date, the IRC channel, the title of the paste, the contents of the paste, and any annotations that may have been added later (I ignored the annotations).

I built a list in memory containing 1000 paste structs with the information listed above. Given that, the following code will allow you to create a Montezuma index on disk:

(defun index-pastes (pastes)
  (let ((index (make-instance 'index
			      :path (merge-pathnames
				     (make-pathname :directory '(:relative "pasteindex"))
				     *corpus-path*)
			      :default-field "contents")))
  (dolist (paste pastes)
    (index-paste index paste))
  (optimize index)
  index))

(defun index-paste (index paste)
  (let ((doc (make-instance 'document)))
    (let ((number   (make-field "id"   (format nil "~S" (paste-number paste))
				:index :untokenized :stored T))
	  (user     (make-field "user"     (paste-user paste)
				:index :untokenized :stored T))
	  (date     (make-field "date"     (date-string (paste-date paste))
				:index :untokenized :stored T))
	  (channel  (make-field "channel"  (paste-channel paste)
				:index :untokenized :stored T))
	  (title    (make-field "title"    (paste-title paste)
				:index :tokenized :stored T))
	  (contents (make-field "contents" (paste-contents paste)
				:stored NIL :index :tokenized)))
      (add-field doc number)
      (add-field doc user)
      (add-field doc date)
      (add-field doc channel)
      (add-field doc title)
      (add-field doc contents)
      (add-document-to-index *paste-index* doc))))

(defun date-string (universal-time)
  (multiple-value-bind (second minute hour date month year)
      (decode-universal-time universal-time)
    (format nil "~D-~2,'0D-~2,'0D ~2,'0D:~2,'0D:~2,'0D"
	    year month date hour minute second)))

Currently, indexing the 1000 pastes (about 2 MB of data) takes about 100 seconds on a 2.4 GHz P4. Once you've done that, you'll want to be able to perform queries against the index.

The search-pastes function below makes it convenient to perform simple queries. You can pass it the field to query along with a query string or list of strings, which will be turned into a query object. A list of strings will become a boolean query where every term must appear. Wildcards (“*”, “?”) can be used in any query terms.

If you pass a pre-built query object as the second argument it will be used unchanged, and the field argument will be ignored.

(defun search-pastes (field query &optional options)
  (etypecase query
    (list
     ;; Make a boolean query where each clause is a wildcard query
     ;; that MUST occur.
     (let ((words query))
       (setf query (make-instance 'boolean-query))
       (dolist (word words)
	 (add-query query
		    (make-instance 'wildcard-query
				   :term (make-term field word))
		    :must-occur))))
    (string
     ;; Make a single-term wildcard query.
     (let ((word query))
       (setf query (make-instance 'wildcard-query
				  :term (make-term field word)))))
    (query
     ;; Don't need to do anything, use it as-is.
     ))
  ;; Perform the search
  (let ((num-results 0))
    (search-each *paste-index* query
		 #'(lambda (doc score)
		     (when (= num-results 0)
		       (format T "~&~5A ~19A ~5A ~15A" "Score" "Date" "#" "User")
		       (format T "~&-------------------------------------------"))
		     (incf num-results)
		     (print-result doc score))
		 options)
    (format T "~&~%~S results displayed." num-results)))

(defun print-result (doc score)
  (let ((paste (get-document *paste-index* doc)))
    (format T "~&~5,2F ~A ~A ~15A~&~vt~A"
	    score
	    (field-data (document-field paste "date"))
	    (field-data (document-field paste "id"))
	    (field-data (document-field paste "user"))
	    10
	    (field-data (document-field paste "title")))))

Here's a simple first example, which finds pastes I pasted:

MONTEZUMA> (search-pastes "user" "lemonodor")
Score Date                #     User           
-------------------------------------------
 6.30 2006-05-08 18:53:03 19817 lemonodor      
          opaque sbcl warning
 6.30 2006-05-05 16:07:11 19726 lemonodor      
          primitive montezuma example
 6.30 2006-04-25 21:07:59 19371 lemonodor      
          flet declarations
 6.30 2006-04-14 12:04:38 18990 lemonodor      
          search-and-replace

4 results displayed.

Now we'll search for pastes with titles containing any word with “sbcl” as a substring, which demonstrates wildcard queries:

MONTEZUMA> (search-pastes "title" "*sbcl*")
Score Date                #     User           
-------------------------------------------
 2.88 2006-04-15 18:19:31 19045 malsyned__     
          My sbclrc
 1.44 2006-04-22 15:37:08 19253 Shine          
          incrementally building sbcl.exe still doesn't work "the party is over"
 1.31 2006-04-21 21:42:52 19239 Shine          
          sbcl backtrace
 1.31 2006-04-20 21:47:40 19212 technomancy    
          sbcl error
 1.05 2006-05-08 18:53:03 19817 lemonodor      
          opaque sbcl warning
 1.05 2006-05-04 23:28:53 19701 interferon     
          SBCL disassembly example
 1.05 2006-05-04 23:28:35 19700 interferon     
          SBCL disassembly example
 1.05 2006-04-30 17:27:51 19551 dabaR          
          Error with SBCL
 1.05 2006-04-27 12:00:59 19428 death          
          sbcl 0.9.11 (room) bugged
 1.05 2006-04-27 03:34:17 19419 arbscht        
          sbcl sarge24 deb failure

10 results displayed.

There are more than 10 pastes that match the above query, but by default only the top 10 results are returned. Let's find all of them:

MONTEZUMA> (search-pastes "title" "*sbcl*" '(:num-docs 1000))
Score Date                #     User           
-------------------------------------------
 2.88 2006-04-15 18:19:31 19045 malsyned__     
          My sbclrc
 1.44 2006-04-22 15:37:08 19253 Shine          
          incrementally building sbcl.exe still doesn't work "the party is over"

[...]

 0.65 2006-04-10 07:24:06 18808 antifuchs      
          gna. nested errors with sbcl 0.9.11.26 and slime
 0.52 2006-04-17 10:12:04 19083 tcr            
          compiling latest SWANK with flags (speed 0) (safety 3) (debug 3) w/ SBCL 0.9.11

22 results displayed.

This query matches pastes containing both “sbcl” and “bug*” from May 2006 on (an example of a range query):

MONTEZUMA> (let ((q (make-instance 'boolean-query)))
	     (add-query q (make-instance 'term-query
					 :term (make-term "contents" "sbcl"))
			:must-occur)
	     (add-query q (make-instance 'wildcard-query
					 :term (make-term "contents" "bug*"))
			:must-occur)
	     (add-query q (make-instance 'range-query
					 :field "date"
					 :lower-term "2006-05"
					 :upper-term nil
					 :include-lower-p T
					 :include-upper-p NIL)
			:must-occur)
	     (search-pastes nil q))
Score Date                #     User           
-------------------------------------------
 0.49 2006-05-09 13:36:33 19832 tritchey       
          this is from a clean build
 0.49 2006-05-09 13:36:33 19832 tritchey       
          this is from a clean build
 0.49 2006-05-09 13:36:33 19832 tritchey       
          this is from a clean build
 0.49 2006-05-09 13:36:33 19832 tritchey       
          this is from a clean build
 0.49 2006-05-09 13:36:33 19832 tritchey       
          this is from a clean build
 0.49 2006-05-09 13:36:33 19832 tritchey       
          this is from a clean build

6 results displayed.

Oops. Looks like there's a bug that's giving us duplicate results. Sorry!

Now look for pastes containing “sbcl” made by someone whose user name does not begin with a “t”:

MONTEZUMA> (let ((q (make-instance 'boolean-query)))
	     (add-query q (make-instance 'term-query
					 :term (make-term "contents" "sbcl"))
			:must-occur)
	     (add-query q (make-instance 'wildcard-query
					 :term (make-term "user" "t*"))
			:must-not-occur)
	     (search-pastes nil q))
Score Date                #     User           
-------------------------------------------
 0.87 2006-04-27 03:34:17 19419 arbscht        
          sbcl sarge24 deb failure
 0.80 2006-04-23 14:38:32 19290 antifuchs      
          mp and fix
 0.75 2006-04-28 11:46:44 19469 Xof            
          see also http://www-jcsu.jesus.cam.ac.uk/~csr21/ed-in-climacs.png
 0.57 2006-04-22 15:37:08 19253 Shine          
          incrementally building sbcl.exe still doesn't work "the party is over"
 0.53 2006-04-26 19:43:19 19408 pfdietz        
          misc.640
 0.53 2006-04-21 18:25:36 19238 waddletron2k   
          my .emacs
 0.53 2006-04-14 16:30:07 18997 jsnell         
          code for lemonodor
 0.52 2006-04-19 17:41:25 19160 slyrus         
          tinaa docs aver any-keyp error
 0.46 2006-04-28 09:02:10 19462 nyef           
          I can't tell if this is progress or not.
 0.46 2006-04-19 15:30:26 19158 Shine          
          problem with SBCL on Windows, compiled with Cygwin

10 results displayed.

For the last example, we'll create a phrase query corresponding to “svn co”:

MONTEZUMA> (defparameter *q*
	     (make-instance 'phrase-query))
*Q*
MONTEZUMA> (add-term-to-query *q* (make-term "contents" "svn"))
#<PHRASE-QUERY field:"contents" terms: "svn":0 >
MONTEZUMA> (add-term-to-query *q* (make-term "contents" "co"))
#<PHRASE-QUERY field:"contents" terms: "svn":0 "co":1 >
MONTEZUMA> (search-pastes nil *q*)
; Evaluation aborted
MONTEZUMA> *q*
#<PHRASE-QUERY field:"contents" terms: "svn":0 "co":1 >
MONTEZUMA> (search-pastes nil *q*)
There is no applicable method for the generic function
  #<STANDARD-GENERIC-FUNCTION CLOSE (32)>
when called with arguments
  (NIL).
   [Condition of type SIMPLE-ERROR]

Well, it seems that despite passing some unit tests, phrase queries are falling down on this larger index. Time to fire up emacs and fix that.

I Still Have a Special Place In My Heart for the 500-in-1 Electronics Kit

Double O Joe also did the painting on this Grand Challenge robot.

As part of the process of figuring out what the hell it is he wants to do, Todd cataloged the things he's done in his life that nobody asked him to do:

• Making a portable "fish caller" circuit, sticking it in a ziploc bag, and trying to catch fish (5th grade)
• Repairing old CB radios and entertaining truckers (5th-6th grade)
• Coding a simple BASIC computer game for the Commodore VIC-20 (6th grade?)
• Creating tiny wearable weird blinking lights with newfangled LEDs and hearing aid batteries (6th grade)
• Coding a LOGO program that would draw out the mission logo of a then-recent space shuttle mission (7th grade)

• Coding a web browser for the Newton PDA (1994)
• Trying to port java to Newton (1995)
• Trying to replicate sonoluminescence experiments (1996)
• Designing and trying to build a propane-powered popcorn popper for Burning Man (1997)
• Coding and building robots at SRL (1998)

What was it with the 70s and the fish caller circuits?

I love the clear line from the experiments done as a kid to the experiments done as an adult. And of course I look at Todd's list and think about what mine would look like, and after a year of freelancing the question of what to do next has started to crop up.

I was wondering the other day if the past year of non-full time, contract work might have saved me from a severe case of burnout. I already was burned out, really. But I know a few other programmers who a year ago were about as dissatisfied and bored as I was, who stayed in the jobs they had and now are in a position of considering a career path having nothing to do with software, or have personal projects they wish they could work on but find they don't have the energy or motivation after a grinding day of coding on something they don't care about. This is not an uncommon problem, of course.

But I think the past year has been restorative in a way that I'm just starting to appreciate. And I'm starting to think I would like to work at a startup again. I think the experience of the freedom of freelancing from home will make me a lot pickier in the future, though.

LispNYC Needs Mentors

LispNYC needs mentors for this year's Google Summer of Code projects.

Looking at the list of suggested projects, I see that “port the Apache Lucene library to lisp” is one. Is it too late to get my $4,500 and a t-shirt?

El Circo Loco

I don't normally get too attached to neighborhood characters. I don't care about the five dollar guy at all. There's a slightly pleasing familiarity in seeing the walking man, but no significant emotional attachment—all he does is walk and read.

The guy everyone called El Circo Loco (whose real name Antoniz Ruiz) was somewhat different. Every time I saw him he was marching and dancing and twirling, in costume. He never just walked, I never saw him sitting, it was non-stop dancing. I supposed he was crazy, but he seemed happy, and it didn't seem completely impossible that he was just some guy who decided one day that life was short and he loved to dance, so that was all he was going to do from now on. You couldn't help but feel a fondness for him (despite the late night whistling that sometimes was annoying).

Two weeks ago El Circo Loco was found dead on the sidewalk a block from my apartment. It was saddening and shocking. He seemed unstoppable.

What really broke my heart was when people in the neighborhood held a memorial in his honor in a small park he often hung out in. I was at Startup School so I couldn't go, but a few days later I walked over and saw a lot of leftover glitter. They even rededicated the fountain to him (it had previously been dedicated to the inventor of the bedpan, which the fountain resembles; ah, hipsters).

Bye, Antonio! I didn't realize how much I'd miss you!

Third DARPA Grand Challenge Announced

On Monday DARPA announced the next Grand Challenge. In this, the third challenge, the desert rats will have to survive in the big city: “Grand Challenge 2005 proved that autonomous ground vehicles can travel significant distances and reach their destination, just as you or I would drive from one city to the next. After the success of this event, we believe the robotics community is ready to tackle vehicle operation inside city limits.”

Some more details on what challenges the robots will or will not face are in the solicitation notice:

At the capstone event, performers will demonstrate a vehicle that exhibits the following behaviors: safe vehicle-following; operation with oncoming traffic; queueing at traffic signals; merging with moving traffic; left-turn across traffic; right-of-way and precedence observance at intersections; proper use of directional signals, brake lights, and reverse lights; passing of moving vehicles; and all the other behaviors implicit in the problem statement. Vehicles will also demonstrate navigation with limited waypoints and safe operation in complex areas such as parking lots. Recognition of traffic signals, street signs or pavement markings is considered outside the scope of the program, as this information will be furnished by DARPA. High-speed highway driving is also outside the scope of the program.

For this challenge DARPA is running two tracks: Track A seems to be the more rigorous track, with team progress tracked against detailed milestones. The bureaucracy appears to approach standard defense contractor levels, but the potential payout is $1,000,000. Track B seems to be about equivalent to the previous challenges in that teams have to supply a video of their vehicle in action and show it off during a site visit by DARPA officials to get in the race. Track B's max award is $100,000. More details are in the Proposer Information Pamphlet.

Both tracks will come together for the qualifying event on October 21, 2007, and will compete together on November 3, 2007:

If you want to compete, note that the participants conference is only about two weeks away!

I guess I'll look into modifying my DARPA GC forum scraper to generate RSS feeds of the new forum.

SubEthaEdit

These notes from Y Combinator's Startup School were taken by about twenty people, simultaneously, using SubEthaEdit. I had never used SubEthaEdit in a group and it was an interesting process.

Each color represents a different contributor. You can see other people typing in realtime, and you can see it when they select text, and you can see where they're moving their cursor.

In the above screenshot, you can see examples where one person has created a template ("# ANNOUNCEMENTS", "# SPEAKER NOTES") and other people are filling it in. People are alternating in entering outline elements, and you can see that sometimes one person will enter a bullet and someone else will fill in some details. You can also see that there may only be a few people making significant contributions at any given time—I think a lot of people just wanted to watch the notes in case they missed something the presenter said.

This screenshot shows some fine grained collaboration: There are people finishing each others sentences, fixing others' typos, and one guy caught the names and company affiliations of the speakers that everyone else missed and made sure all the answers were properly credited.

A couple times I saw some neat stuff happen. For example, during Joshua Schacter's talk, someone had missed the beginning of a new section and was putting notes under the old heading. I selected the line he was working on, he saw that and stopped typing, i cut and pasted into the correct heading, and he resumed typing (I had been wondering why SubEthaEdit showed other people's text selections, and this perhaps illustrated the issue).

What worked well:

• One or two people with previous SubEthaEdit experience laid out a decent document template to follow, with an area for real-time chat, per-speaker sections and a list of contributors.
• People did a pretty good job of filling in pieces that other people missed.
• SubEthaEdit's Bonjour support made it trivial to find the document once you got on the wifi network.
• Markdown let us create a document that looks OK in both raw text and HTML (I posted a Markdown reference and cheat sheet using OS X's “Web Sharing” which was available via a link-local hostname that didn't require a DNS server or anything on our ad hoc network).

What didn't work as well:

• As far as I know SubEthaEdit is only available for OS X, so no Mac means no note sharing.
• Author information isn't persistent—if you save the file and reload it, you lose the color coded record of who wrote what. This happened across lunch, for example.
• In the list above I mentioned “the document” and “the wifi network”, but in reality we had three different wifi networks, two of which were named “SubEthaEdit” and “Subetha Group”, and each network had people running SubEthaEdit and working on different documents. It took a while to mostly converge on a sorta canonical document, and I'm not sure that there was a complete convergence.
• There was no head editor, and the final notes are something that Frankenstein might be scared of. I don't think any of us actually knew Markdown, and my attempts to learn quickly and work the document over into Markdown format as we went were only partially successful.

We did post the notes frequently during the day to a web page, and doing that required an effort that was either Herculean or Sisyphusian. Dan Moniz brought a wifi access point with an EVDO card for internet access (Stanford's wifi network is closed, so we had no internet that way), but it was kind of flaky and we didn't want everyone to be affected by the unreliability. So I stayed on the SubEthaEdit ad hoc network and helped edit the notes, periodically saving to my Public folder. Dan would then use Bluetooth File Exchange to grab the notes from my machine and post them to the web through his wifi AP with EVDO backhaul. Periodically I would use my cell phone as a GPRS modem via Bluetooth to grab something I needed online. I was about to say that this was probably not the geekiest thing happening at the conference, but upon reflection I think it is in fact a strong contender for the top slot.

Startup School

This weekend was Y Combinator's Startup School. My friend Dave, also an ex-Evo'er, and I drove up from LA to attend.

Maybe the most interesting thing about Startup School was that I was directly faced with the fact that these guys started companies. Nothing magical, they just did it. And most will fail, but that's how you learn, and it's important not to be afraid of failure.

Other than that, some of the speakers were interesting (I wouldn't have guessed it, but the lawyer was great) and some were not (is a presentation about how awesome Google is supposed to convince me to not start my own company?). Paul Graham has a new essay based on his talk from this Saturday, “The Hardest Lessons for Startups to Learn”.

There are notes online, but they might only be interesting if you were there and missed something in your own note taking.

You can find lots of pictures online of Startup School. I have some photos of the surrounding events, but if you've seen one conference with entrepreneurial nerds backed by powerpoint you've seen them all (though you might find some of Trevor Blackwell's robots interesting).

The blog posts are trickling in, along with the articles. (Jesus,) there are even (fucking) podcasts.

Between the Y Combinator reception, Startup School and the Lisp brunch organized by Dan Moniz I met a lot of interesting people and lemonodor stalkers. Actually it's great to meet lemonodor fans—thanks, guys. And I got to see a few friends (Gavin, est, dnm, Joshua) I don't get to see often. But holy shit am I tired now, and glad to be home.