Last week Lori, Carrie and I were extras in a friend's short film. We played indie rockers watching a band in a bar. Standing in a hazy club with a bunch of hipsters, beer in hand and nodding to some guitar-based rock and/or roll is not exactly a foreign concept, but doing it as simulation was an odd way to recall that I haven't actually done it for real in a while and that I miss it.
This was the first time I've participated first-hand in the stereotypical Los Angeles industry as opposed to just observing. Maybe I should take advantage of the free time I have these days and have some fun as an extra. Next stop on the background actor train, a Hollywood blockbuster [via robotwisdom]:
Before Take 7 Petersen comes on. “One more,” he says. “Now with full acting. Let's make this the one!” It works. As the first AD calls “Action,” it's like the entire background cast has been hit with a Taser. The place goes berserk, fear and chaos cranked to 11. And I'm swept up in it.
Too bad I didn't think of this while they were shooting Miami Vice.
“This crash highlights the safety concerns we've voiced all along,” said Andy Cebula, AOPA executive vice president of government affairs. “The FAA should not permit UAV operations until they are certified to the same level of safety as manned aircraft.”
“Thankfully, in this accident no one in the air or on the ground was hurt,“ Cebula added. ”But just think that if a pilot had been flying legally under the TFR and the UAV hit the aircraft from behind and above -- the pilot would have had no chance to see and avoid the uncontrolled UAV.”
The Aircraft Owners and Pilots Association wasn't too happy about the UAVs before they crashed, either, since the FAA had imposed flight restrictions on a 350 mile section of Arizona and New Mexico because of the Predators' operations. And in general the AOPA worries about the danger UAVs pose to other aircraft.
As it turns out, the preliminary NTSB report on the accident seems to indicate that the crash was due to pilot error:
The pilot reported that during the flight the console at PPO-1 "locked up", prompting him to switch control of the UAV to PPO-2. Checklist procedures state that prior to switching operational control between the two consoles, the pilot must match the control positions on the new console to those on the console, which had been controlling the UAV. The pilot stated in an interview that he failed to do this. The result was that the stop/feather control in PPO-2 was in the fuel cutoff position when the switch over from PPO-1 to PPO-2 occurred. As a result, the fuel was cut off to the UAV when control was transferred to PPO-2.
The pilot stated that after the switch to the other console, he noticed the UAV was not maintaining altitude but did not know why. As a result he decided to shut down the GCS so that the UAV would enter its lost link procedure, which called for the UAV to climb to 15,000 feet above mean sea level and to fly a predetermined course until contact could be established. With no engine power, the UAV continued to descend below line-of-site communications and further attempts to re-establish contact with the UAV were not successful
The U.S. Border Patrol isn't the only group flying UAVs in the area. A vigilante group called the American Border Patrol, in conjunction with the Minutemen, has been flying their own homebuilt UAV—apparently a souped up model airplane. They even have video of their vehicle in action.
And then there is the anti-vigilante group UAV flown near the border by Chris Csikszentmihalyi. Quite the crowded airspace down there these days.
I decided to spend a little time optimizing Montezuma indexing for speed, with the hope that I could get it to index the Planet Lisp archives within a reasonable amount of time.
Indexing 1000 pastes took about 100 seconds, as I mentioned the other day. Avoiding 6 million garbage-generating calls to subseq saved 10 seconds. Cutting down on the number of calls to string< and string> saved 4 seconds. Each was a significant speedup, but nothing dramatic.
After doing some of this benchmarking, the laptop I was using overheated and shut down. Since this time there were no cats lying on top and therefore no obvious scapegoats, I thought maybe the vents had just gotten clogged and so I blasted the hell out of it with canned air.
The comprehensive air blast cleaning cut 45 seconds off the benchmark time. It was like getting a processor upgrade.
In addition to the public appearances, the whole family will be blogging our experiences on the road and interviewing comics artists around the country about their work. We plan to have a lot of fun (when we're not going crazy or trying to kill each other) and we hope you'll be part of it all when we come to your state.
Yesterday I had one cat-induced thermal shutdown of that machine, plus one ctrl-paw-delete.
Montezuma is coming along pretty well. Indexing is complete and seems mostly bug free, and Gary King and I are finishing off the support for searches. We've got simple term queries, phrase queries, wildcard queries, range queries and boolean queries pretty much working.
For testing, I grabbed 1000 pastes from the paste service at paste.lisp.org. Each paste consists of a number (the unique paste ID), the user who made the paste, the date, the IRC channel, the title of the paste, the contents of the paste, and any annotations that may have been added later (I ignored the annotations).
I built a list in memory containing 1000 paste structs with the information listed above. Given that, the following code will allow you to create a Montezuma index on disk:
(defun index-pastes (pastes) (let ((index (make-instance 'index :path (merge-pathnames (make-pathname :directory '(:relative "pasteindex")) *corpus-path*) :default-field "contents"))) (dolist (paste pastes) (index-paste index paste)) (optimize index) index)) (defun index-paste (index paste) (let ((doc (make-instance 'document))) (let ((number (make-field "id" (format nil "~S" (paste-number paste)) :index :untokenized :stored T)) (user (make-field "user" (paste-user paste) :index :untokenized :stored T)) (date (make-field "date" (date-string (paste-date paste)) :index :untokenized :stored T)) (channel (make-field "channel" (paste-channel paste) :index :untokenized :stored T)) (title (make-field "title" (paste-title paste) :index :tokenized :stored T)) (contents (make-field "contents" (paste-contents paste) :stored NIL :index :tokenized))) (add-field doc number) (add-field doc user) (add-field doc date) (add-field doc channel) (add-field doc title) (add-field doc contents) (add-document-to-index *paste-index* doc)))) (defun date-string (universal-time) (multiple-value-bind (second minute hour date month year) (decode-universal-time universal-time) (format nil "~D-~2,'0D-~2,'0D ~2,'0D:~2,'0D:~2,'0D" year month date hour minute second)))
Currently, indexing the 1000 pastes (about 2 MB of data) takes about 100 seconds on a 2.4 GHz P4. Once you've done that, you'll want to be able to perform queries against the index.
The search-pastes function below makes it convenient to perform simple queries. You can pass it the field to query along with a query string or list of strings, which will be turned into a query object. A list of strings will become a boolean query where every term must appear. Wildcards (“*”, “?”) can be used in any query terms.
If you pass a pre-built query object as the second argument it will be used unchanged, and the field argument will be ignored.
(defun search-pastes (field query &optional options) (etypecase query (list ;; Make a boolean query where each clause is a wildcard query ;; that MUST occur. (let ((words query)) (setf query (make-instance 'boolean-query)) (dolist (word words) (add-query query (make-instance 'wildcard-query :term (make-term field word)) :must-occur)))) (string ;; Make a single-term wildcard query. (let ((word query)) (setf query (make-instance 'wildcard-query :term (make-term field word))))) (query ;; Don't need to do anything, use it as-is. )) ;; Perform the search (let ((num-results 0)) (search-each *paste-index* query #'(lambda (doc score) (when (= num-results 0) (format T "~&~5A ~19A ~5A ~15A" "Score" "Date" "#" "User") (format T "~&-------------------------------------------")) (incf num-results) (print-result doc score)) options) (format T "~&~%~S results displayed." num-results))) (defun print-result (doc score) (let ((paste (get-document *paste-index* doc))) (format T "~&~5,2F ~A ~A ~15A~&~vt~A" score (field-data (document-field paste "date")) (field-data (document-field paste "id")) (field-data (document-field paste "user")) 10 (field-data (document-field paste "title")))))
Here's a simple first example, which finds pastes I pasted:
MONTEZUMA> (search-pastes "user" "lemonodor") Score Date # User ------------------------------------------- 6.30 2006-05-08 18:53:03 19817 lemonodor opaque sbcl warning 6.30 2006-05-05 16:07:11 19726 lemonodor primitive montezuma example 6.30 2006-04-25 21:07:59 19371 lemonodor flet declarations 6.30 2006-04-14 12:04:38 18990 lemonodor search-and-replace 4 results displayed.
Now we'll search for pastes with titles containing any word with “sbcl” as a substring, which demonstrates wildcard queries:
MONTEZUMA> (search-pastes "title" "*sbcl*") Score Date # User ------------------------------------------- 2.88 2006-04-15 18:19:31 19045 malsyned__ My sbclrc 1.44 2006-04-22 15:37:08 19253 Shine incrementally building sbcl.exe still doesn't work "the party is over" 1.31 2006-04-21 21:42:52 19239 Shine sbcl backtrace 1.31 2006-04-20 21:47:40 19212 technomancy sbcl error 1.05 2006-05-08 18:53:03 19817 lemonodor opaque sbcl warning 1.05 2006-05-04 23:28:53 19701 interferon SBCL disassembly example 1.05 2006-05-04 23:28:35 19700 interferon SBCL disassembly example 1.05 2006-04-30 17:27:51 19551 dabaR Error with SBCL 1.05 2006-04-27 12:00:59 19428 death sbcl 0.9.11 (room) bugged 1.05 2006-04-27 03:34:17 19419 arbscht sbcl sarge24 deb failure 10 results displayed.
There are more than 10 pastes that match the above query, but by default only the top 10 results are returned. Let's find all of them:
MONTEZUMA> (search-pastes "title" "*sbcl*" '(:num-docs 1000)) Score Date # User ------------------------------------------- 2.88 2006-04-15 18:19:31 19045 malsyned__ My sbclrc 1.44 2006-04-22 15:37:08 19253 Shine incrementally building sbcl.exe still doesn't work "the party is over" [...] 0.65 2006-04-10 07:24:06 18808 antifuchs gna. nested errors with sbcl 0.9.11.26 and slime 0.52 2006-04-17 10:12:04 19083 tcr compiling latest SWANK with flags (speed 0) (safety 3) (debug 3) w/ SBCL 0.9.11 22 results displayed.
This query matches pastes containing both “sbcl” and “bug*” from May 2006 on (an example of a range query):
MONTEZUMA> (let ((q (make-instance 'boolean-query))) (add-query q (make-instance 'term-query :term (make-term "contents" "sbcl")) :must-occur) (add-query q (make-instance 'wildcard-query :term (make-term "contents" "bug*")) :must-occur) (add-query q (make-instance 'range-query :field "date" :lower-term "2006-05" :upper-term nil :include-lower-p T :include-upper-p NIL) :must-occur) (search-pastes nil q)) Score Date # User ------------------------------------------- 0.49 2006-05-09 13:36:33 19832 tritchey this is from a clean build 0.49 2006-05-09 13:36:33 19832 tritchey this is from a clean build 0.49 2006-05-09 13:36:33 19832 tritchey this is from a clean build 0.49 2006-05-09 13:36:33 19832 tritchey this is from a clean build 0.49 2006-05-09 13:36:33 19832 tritchey this is from a clean build 0.49 2006-05-09 13:36:33 19832 tritchey this is from a clean build 6 results displayed.
Oops. Looks like there's a bug that's giving us duplicate results. Sorry!
Now look for pastes containing “sbcl” made by someone whose user name does not begin with a “t”:
MONTEZUMA> (let ((q (make-instance 'boolean-query))) (add-query q (make-instance 'term-query :term (make-term "contents" "sbcl")) :must-occur) (add-query q (make-instance 'wildcard-query :term (make-term "user" "t*")) :must-not-occur) (search-pastes nil q)) Score Date # User ------------------------------------------- 0.87 2006-04-27 03:34:17 19419 arbscht sbcl sarge24 deb failure 0.80 2006-04-23 14:38:32 19290 antifuchs mp and fix 0.75 2006-04-28 11:46:44 19469 Xof see also http://www-jcsu.jesus.cam.ac.uk/~csr21/ed-in-climacs.png 0.57 2006-04-22 15:37:08 19253 Shine incrementally building sbcl.exe still doesn't work "the party is over" 0.53 2006-04-26 19:43:19 19408 pfdietz misc.640 0.53 2006-04-21 18:25:36 19238 waddletron2k my .emacs 0.53 2006-04-14 16:30:07 18997 jsnell code for lemonodor 0.52 2006-04-19 17:41:25 19160 slyrus tinaa docs aver any-keyp error 0.46 2006-04-28 09:02:10 19462 nyef I can't tell if this is progress or not. 0.46 2006-04-19 15:30:26 19158 Shine problem with SBCL on Windows, compiled with Cygwin 10 results displayed.
For the last example, we'll create a phrase query corresponding to “svn co”:
MONTEZUMA> (defparameter *q* (make-instance 'phrase-query)) *Q* MONTEZUMA> (add-term-to-query *q* (make-term "contents" "svn")) #<PHRASE-QUERY field:"contents" terms: "svn":0 > MONTEZUMA> (add-term-to-query *q* (make-term "contents" "co")) #<PHRASE-QUERY field:"contents" terms: "svn":0 "co":1 > MONTEZUMA> (search-pastes nil *q*) ; Evaluation aborted MONTEZUMA> *q* #<PHRASE-QUERY field:"contents" terms: "svn":0 "co":1 > MONTEZUMA> (search-pastes nil *q*) There is no applicable method for the generic function #<STANDARD-GENERIC-FUNCTION CLOSE (32)> when called with arguments (NIL). [Condition of type SIMPLE-ERROR]
Well, it seems that despite passing some unit tests, phrase queries are falling down on this larger index. Time to fire up emacs and fix that.
As part of the process of figuring out what the hell it is he wants to do, Todd cataloged the things he's done in his life that nobody asked him to do:
What was it with the 70s and the fish caller circuits?
I love the clear line from the experiments done as a kid to the experiments done as an adult. And of course I look at Todd's list and think about what mine would look like, and after a year of freelancing the question of what to do next has started to crop up.
I was wondering the other day if the past year of non-full time, contract work might have saved me from a severe case of burnout. I already was burned out, really. But I know a few other programmers who a year ago were about as dissatisfied and bored as I was, who stayed in the jobs they had and now are in a position of considering a career path having nothing to do with software, or have personal projects they wish they could work on but find they don't have the energy or motivation after a grinding day of coding on something they don't care about. This is not an uncommon problem, of course.
But I think the past year has been restorative in a way that I'm just starting to appreciate. And I'm starting to think I would like to work at a startup again. I think the experience of the freedom of freelancing from home will make me a lot pickier in the future, though.
Looking at the list of suggested projects, I see that “port the Apache Lucene library to lisp” is one. Is it too late to get my $4,500 and a t-shirt?
I don't normally get too attached to neighborhood characters. I don't care about the five dollar guy at all. There's a slightly pleasing familiarity in seeing the walking man, but no significant emotional attachment—all he does is walk and read.
The guy everyone called El Circo Loco (whose real name Antoniz Ruiz) was somewhat different. Every time I saw him he was marching and dancing and twirling, in costume. He never just walked, I never saw him sitting, it was non-stop dancing. I supposed he was crazy, but he seemed happy, and it didn't seem completely impossible that he was just some guy who decided one day that life was short and he loved to dance, so that was all he was going to do from now on. You couldn't help but feel a fondness for him (despite the late night whistling that sometimes was annoying).
Two weeks ago El Circo Loco was found dead on the sidewalk a block from my apartment. It was saddening and shocking. He seemed unstoppable.
What really broke my heart was when people in the neighborhood held a memorial in his honor in a small park he often hung out in. I was at Startup School so I couldn't go, but a few days later I walked over and saw a lot of leftover glitter. They even rededicated the fountain to him (it had previously been dedicated to the inventor of the bedpan, which the fountain resembles; ah, hipsters).
Bye, Antonio! I didn't realize how much I'd miss you!
On Monday DARPA announced the next Grand Challenge. In this, the third challenge, the desert rats will have to survive in the big city: “Grand Challenge 2005 proved that autonomous ground vehicles can travel significant distances and reach their destination, just as you or I would drive from one city to the next. After the success of this event, we believe the robotics community is ready to tackle vehicle operation inside city limits.”
Some more details on what challenges the robots will or will not face are in the solicitation notice:
At the capstone event, performers will demonstrate a vehicle that exhibits the following behaviors: safe vehicle-following; operation with oncoming traffic; queueing at traffic signals; merging with moving traffic; left-turn across traffic; right-of-way and precedence observance at intersections; proper use of directional signals, brake lights, and reverse lights; passing of moving vehicles; and all the other behaviors implicit in the problem statement. Vehicles will also demonstrate navigation with limited waypoints and safe operation in complex areas such as parking lots. Recognition of traffic signals, street signs or pavement markings is considered outside the scope of the program, as this information will be furnished by DARPA. High-speed highway driving is also outside the scope of the program.
For this challenge DARPA is running two tracks: Track A seems to be the more rigorous track, with team progress tracked against detailed milestones. The bureaucracy appears to approach standard defense contractor levels, but the potential payout is $1,000,000. Track B seems to be about equivalent to the previous challenges in that teams have to supply a video of their vehicle in action and show it off during a site visit by DARPA officials to get in the race. Track B's max award is $100,000. More details are in the Proposer Information Pamphlet.
Both tracks will come together for the qualifying event on October 21, 2007, and will compete together on November 3, 2007:
If you want to compete, note that the participants conference is only about two weeks away!
I guess I'll look into modifying my DARPA GC forum scraper to generate RSS feeds of the new forum.
Each color represents a different contributor. You can see other people typing in realtime, and you can see it when they select text, and you can see where they're moving their cursor.
In the above screenshot, you can see examples where one person has created a template ("# ANNOUNCEMENTS", "# SPEAKER NOTES") and other people are filling it in. People are alternating in entering outline elements, and you can see that sometimes one person will enter a bullet and someone else will fill in some details. You can also see that there may only be a few people making significant contributions at any given time—I think a lot of people just wanted to watch the notes in case they missed something the presenter said.
This screenshot shows some fine grained collaboration: There are people finishing each others sentences, fixing others' typos, and one guy caught the names and company affiliations of the speakers that everyone else missed and made sure all the answers were properly credited.
A couple times I saw some neat stuff happen. For example, during Joshua Schacter's talk, someone had missed the beginning of a new section and was putting notes under the old heading. I selected the line he was working on, he saw that and stopped typing, i cut and pasted into the correct heading, and he resumed typing (I had been wondering why SubEthaEdit showed other people's text selections, and this perhaps illustrated the issue).
What worked well:
What didn't work as well:
We did post the notes frequently during the day to a web page, and doing that required an effort that was either Herculean or Sisyphusian. Dan Moniz brought a wifi access point with an EVDO card for internet access (Stanford's wifi network is closed, so we had no internet that way), but it was kind of flaky and we didn't want everyone to be affected by the unreliability. So I stayed on the SubEthaEdit ad hoc network and helped edit the notes, periodically saving to my Public folder. Dan would then use Bluetooth File Exchange to grab the notes from my machine and post them to the web through his wifi AP with EVDO backhaul. Periodically I would use my cell phone as a GPRS modem via Bluetooth to grab something I needed online. I was about to say that this was probably not the geekiest thing happening at the conference, but upon reflection I think it is in fact a strong contender for the top slot.
This weekend was Y Combinator's Startup School. My friend Dave, also an ex-Evo'er, and I drove up from LA to attend.
Maybe the most interesting thing about Startup School was that I was directly faced with the fact that these guys started companies. Nothing magical, they just did it. And most will fail, but that's how you learn, and it's important not to be afraid of failure.
Other than that, some of the speakers were interesting (I wouldn't have guessed it, but the lawyer was great) and some were not (is a presentation about how awesome Google is supposed to convince me to not start my own company?). Paul Graham has a new essay based on his talk from this Saturday, “The Hardest Lessons for Startups to Learn”.
There are notes online, but they might only be interesting if you were there and missed something in your own note taking.
You can find lots of pictures online of Startup School. I have some photos of the surrounding events, but if you've seen one conference with entrepreneurial nerds backed by powerpoint you've seen them all (though you might find some of Trevor Blackwell's robots interesting).
Between the Y Combinator reception, Startup School and the Lisp brunch organized by Dan Moniz I met a lot of interesting people and lemonodor stalkers. Actually it's great to meet lemonodor fans—thanks, guys. And I got to see a few friends (Gavin, est, dnm, Joshua) I don't get to see often. But holy shit am I tired now, and glad to be home.