I've seen the first thing more than once (Lisp prototype, Java/C++ follow-up fiasco). In one case, I was part of the follow-up fiasco; in another I was an abortive interviewee for a fiasco that was planned but unfortunately never happened.

I haven't seen the second thing myself (product developed in Java/C++, serendipitous port to Lisp, everyone happier). I could believe it happens, and I could also believe that it doesn't happen.

I agree that I have definitely never seen the third thing (prototype developed in Lisp, follow-on Java/C++ product that is better). This sort of seems like a variant of the first thing, though.

There's a fourth possible thing, not mentioned: Lisp prototype, which becomes Lisp final product. I haven't actually seen much of this either. (Here's where I'll get the flames, but let's at least raise the bar for them: discount emacs, discount academic projects, discount any project paid for by DARPA/ONR/NASA (without subsequent commercial recoup), and finally discount any Lisp protototype application whose purchasers are currently furiously engaged in a fiasco of a Java/C++ port.)

The original poster seemed to believe that he was making an argument in favor of Lisp by pointing out that it's damnedly difficult to port Lisp programs to other languages. But there are two ways to take this:

1) As target implementation languages, Java and C++ suck because it's hard to map Lisp prototype applications onto them.

2) As a source prototyping language, Lisp sucks because it's hard to map it onto Java and C++ implementations.

Given the current installed base of Java, C++, and Lisp, which argument is more likely to matter?

So let's dispense for the moment with Lisp as a prototyping language for Java/C++. What I really want to know is: why has the route of Lisp (as prototyping language) to Lisp (as final application language) had such little success, especially given that anyone who really uses Lisp gets blown away by expressivity and ease of development? (The only non-explanation I'd like to avoid is "People are stupid!", because those very same people were presumably just as stupid while Java and C++ took over. Why did they succeed where Lisp didn't?)

Some possibilities (a.k.a "wild speculation with no basis in experience")

* delivering 'shrinkwrap' apps in a commercial Lisp is often a little
messy: either technically or financially (runtime licensing). Note
that this is much less the case, for, say, MCL than it is for Unix
lisps - which is probably why the Mac has more shrinkwrap apps
developed in Lisp. See e.g. http://www.noteheads.com/ - but I'm not
a Machead and someone who actually uses that platform can give you
other examples (Rainer?)

* inhouse or server-based apps are usually only commissioned by
corporations, who (again, generalising wildly) often have standards
that prohibit the use of "unapproved" software or platforms. Because Lisp is always billed as "good for the hard problems", there's little incentive for anyone to try it on the easy or small problems, so when a proposal is made to use it, nobody other than the project champion has any familarity with it other than what they vaguely remember from college, and they'd rather stick to what they know.

Besides,
it's a lot harder to find out about this internal-only stuff unless
you know someone internal.

* besides, the attraction of lisp to the developers is the interactive
thing. As soon as you shinkwrap you probably have to disable the repl
(or at least hide it somewhere) and where's the fun in that ;-)

Sorry about the formatting here. Half of it has line breaks where emacs wanted to put them for 70 column viewing, and the rest was typed directly into this infernal narrow window so probably has automatic wrapping at page width.

From ELUG 98:

- prototype in Common Lisp
- prototype ready ahead of time -> time to
add extra features
- get carried away with adding features, so
management doesn't notice the deadline getting
closer
- deadline arrives -> ship Common Lisp product

If I remember correctly it was even some military
project.

Not having ever gone from a CL -> C++ implementation -- I still have a hard time believing it. Lispy implementations have a real advantage of reducing the cost (in time) of trying different things: so for a slippery domain, they're great.

That being said, once you've got an implementation nailed down and you get the order "Ok, do it in C++/C/Fortran/ASM/A Turing Machine", how hard can it be? Unlike working from a spec, you've actually got a clear, concise description of what's desired. I don't know, maybe they're hiring the wrong grunt programmers.

I've seen both (1) and (2) happen on the same project. The project was developed in Lisp by 50 non-Lisp-programmers, and had really cool features but was slow. So management mandated that it be redone in C++ (but a manager one step down carved out 5 Lisp programmers to redo it in Lisp). The big team didn't try to port the project; they redid it from the ground up in C++, inventing an interpreter for a new dynamic scripting language to do the job of the Lisp code. The small team started over in Lisp. The Lisp version's UI was faster than the C++ version, and it had most of the features of the C++ version, plus novel features the C++ version didn't have. However, the Lisp version was cancelled, according to the decision makers, because the C++ version represented a larger investment.

"However, the Lisp version was cancelled, according to the decision
makers, because the C++ version represented a larger investment."

Augh. Isn't that an example of exactly what MBAs are supposed to
learn to avoid? Don't they know about sunk cost?

Here's my wacky explanation for the Lisp to C++ life cycle: People
(arguably, even people here) consider Lisp the language to use for
working on hard problems, for the basic reasons why we like Lisp in
the first place -- it's more flexible, more expressive, and all that.
But once the prototype is finished, the problem can be said to be
sufficiently understood. I claim that it's a general phenomenon that
once a hard problem is understood, it's suddenly no longer a "hard
problem". And if it's not hard, then (says management) there's no
reason not to rewrite the program in C++, Java, Perl, what have you,
because you know what the process is supposed to be, and C/J/P
programmers are easier to replace than Lisp programmers, in case your
development group moves on.

This is a generalization of why I think AI has a problem getting
respect. Once a problem in AI is solved sufficiently well, it's
demoted from being an AI problem to just being an algorithm.
Computers playing chess isn't "artificial intelligence", it's just a
search problem. Machine translation of one language to another isn't
"artificial intelligence", it's just a representation issue which also
requires access to an enormous linguistic corpus. Robot vaccuum
cleaners certainly aren't artificially intelligent, they just
implement certain behaviors in a subsumption architecture. And so on.

Thoughts?

I was trying to compress a fairly complex set of events and issues into a short remark, and I did violence to some important considerations.

The fact is that in the review meeting where the decision was made on whether to continue or discontinue the lisp project (which was an OS project, by the way--very interesting work), we were indeed told that the lisp version was good (the reviewers even used the words 'better' and 'superior,' though two out of three of them were on the C++ team), and that the reason they were terminating our effort was that the company could not afford to continue both projects and that they wanted to try to realize return on the one with more investment in it. But there was more to it than that.

Part of it, I'm sure, was that they didn't want to approach a large and heavily-stressed group of very smart engineers and say "abandon your very hard work AGAIN and switch over to this other project" (they did that to us, of course, but there were only five of us--much less disruptive). Another point is that the machine was very resource-constrained and we were pushing the available RAM; we were faster, but we were also larger, and they judged that the lower speed of the C++ code (along with some other problems, notably a texbook case of the brittle class problem) was acceptable in trade for the greater free space it left. Finally, the lisp in question was itself under development and lacked a couple of important features (notably native threads support). It looked like they would have come along in time for the release targets, but it would have been close; it made people very nervous.

It's not clear, therefore, that the decision they made was wrong, though it certainly was disappointing. As it turns out, the machine in question was not a success in the market, and I don't think the lisp-based OS would have made the difference, though of course all of us working on it sure wanted to try it.

Mikel, I guess I have one of those (later) machines here. It is quite nice.

What happened with the source of the Lisp-based OS?

Nimrod said:
"Not having ever gone from a CL -> C++ implementation -- I still have a hard time believing it. [...] once you've got an implementation nailed down and you get the order "Ok, do it in C++/C/Fortran/ASM/A Turing Machine", how hard can it be?"

I know that this is the standard story about why Lisp _should_ be a great prototyping language, but I'd rather pay attention to even anecdotal evidence than insist that the ideology indicates that the evidence must be wrong. All the anecdotal evidence I've seen about Lisp -> C++/Java (a couple of them being personal anecdotes) indicates that it doesn't work out very well, and I've never seen a counterexample.

I really think that there are two different forces at work with Lisp and prototyping:

1) aspects of Lisp that just allow trying different things out quickly (say, REPL), and

2) aspects of the language that encourage a deeply different program structure than the target language supports easily (say, closures).

In between are things like macrology --- it's not rocket science to cash out your macros in C or Turing machines or lex/yacc, but it's not easy either. (If it were easy, then why would we all be so enthusiastic about macros?)

Aspects of Lisp in category #1 let you explore the implementation space faster (and if so, shouldn't contribute to the costs of the eventual port of the worked-out idea). Aspects in category #2 make the port harder by encouraging you to race ahead with features because it's so easy. This is like a round-trip hike where the first half is downhill, which makes you want to hike a long way --- the very thing that makes the first thing (Lisp dev) easy is what will make the second thing (the port) harder.

Anyway, why should Lisp people _settle_ for the marginalized role of a prototyping language, especially now that we have scripting languages like Perl and Python which can also play that role?

Rainer:

The machines were really cool to play with, and there were bigger and more capable versions of them internally that never made it to product phase. (It was a completely new product category and there was a lot of guesswork about what people would be willing to pay for these things. There were prototypes that would have cost many thousands of dollars, but the goal near the end was to try to get it below $500.)

I know of archives of the source code that have survived (it's been ten years since the code was compiled and booted, I think), and I approached the company that owns it about open-sourcing it, but they say they only do that with code that is currently in development.

Timboy:

Besides the OS project I worked on two other large, multi-year projects written in Lisp, each of which garnered acclaim for technical successes, each of which was also discontinued. It is actually hard to port a large lisp project to a language like C or C++ (it's arguably easier to write a compiler that compiles lisp to C). One reason is that you write a big lisp project by simultaneously decomposing a toplevel idea into smaller logical pieces and building up the underlying language to make it convenient to represent those toplevel ideas. You build easy, flexible data structures and try stuff out a bunch of different ways until it works and makes sense, then you subsititute different data structures that are more efficient in order to get better performance. This all happens by continuous iteration and constant refactoring, with the code working the whole time.

Then it's time to look at moving the whole thing to C and the entire character of the project changes. Suddenly all the iteration and refactoring must stop dead because you have to freeze the project so that it doesn't change out from under the port. And you have to figure out how to represent the data structures that you evolved in Lisp using the C (or C++, or whatever). In the Lisp code these will be simple and natural because you have evolved the language to represent them naturally, but the simplest of them may be very complicated to represent in C. At each stage of the port you find yourself going, "well, we're going to need a library to represent interned strings for this part, and we'll need to build a hashing library for this part that can handle all these different primitive types correctly, and we'll need another library to represent these expressions which will require a complicated set of control structures and caching strategies for traversal to work efficiently, and..." And the whole time you're planning out and decomposing and implementing and testing all this infrastructure, your project isn't progressing, features are not being implemented, bugs aren't being fixed (except in the new infrastructure code), and you've gone from iterating several times a day on the design while the whole thing runs, to days or weeks before you can even run the thing to see if you got the basic infrastructure right.

It's not easy. I know thee feeling that comes on you when you contemplate moving a big lisp project into (for example) C code: it's sort of "Oooohhh..." with a sinking sensation and downcast eyes. "Okay, this is going to suck."

Timboy --

We're in agreement.

Timboy:

Why were the Lisp projects cancelled?

The first one was a system for testing system software against its specification. It successfully discovered through automated probing thousands of bugs in a forthcoming and strategic release of system software. It was hailed as a great success. There was at the time another testing product that was a sort of scripting language for making software robots that acted like virtual users. They were successful too. The two systems were very different, and both very successful. They were seen by some as competitive, and the scripting solution had users. Our solution had only us, the developers. It found more bugs, but when software projects compete the ones that have users beat the ones that don't. Ours was cancelled because it was seen as competitive with another system that had more users.

Then came the OS project about which I've already written. (It was Apple's Newton; I didn't mention it by name because past experience suggests that the conversation will veer sharply off-topic when I do.) Our lisp-based system was cancelled because it was in competition with a C++ system that represented the company's main bet.

Next I worked on SK8. SK8 was an authoring system of amazing power. Lots of people have called it 'HyperCard on steroids' which sort of hints at what it was like. Basically SK8 was an amazing technology in search of a market. It was cancelled because Apple couldn't see a clear way to 'productize' it. You could make anything with it quickly and easily, but Apple couldn't decide how to package it so that people would want to spend money on it. It was cancelled because there was no clear product strategy.

If there is a lesson to be taken from my experiences, I think it's that technology per se does not decide the success or failure of engineering projects. Language is just one of many factors that contribute to a project's success or failure. In each of these cases Lisp contributed significantly to the project's successes, and probably wasn't responsible for its eventual failure. It's worth remembering that most software projects fail; they are always big, complicated, expensive gambles.

Atleast the Sk8 sources were able to escape. Usually the code is lost...

What about Allegro's magical lisp->java products. It looks like these allow you to develop in Lisp and deploy in Java. Anyone have (or heard about) experiences using these?

Where did the sk8 sources escape to? I've never been able to find them.

steve jenson:

SK8 and its sources are on:

ftp://ftp.apple.com/research/sk8