An interview with John Walker - Part 2

This is the second part of the series documenting my recent interview with John Walker. Where the first part looked at the early history of the company, this part focuses more on the architectural evolution of AutoCAD and its APIs. I have not linked to terms previously linked to in the first part of the series, which should also be referred to for typographical conventions, if they're not clear.

AutoCAD's architecture & APIs

We've talked a little about Variables and Expressions, already. What was behind the decision to use Lisp as a language?

Well, that's interesting. I would say, as much as anything, opportunity. We had this menu macro language, and that clearly had gone as far as it could go. We needed something that would provide access to the database. We needed something that could actually do algorithmic operations needed in writing the program. And it had to fit in 64K. It had to fit in a 64K segment, basically, and it had to fit - with that 64K segment - in a 640K IBM PC environment. Along with all the rest of AutoCAD, which at that point was hideously overlaid just to squeeze AutoCAD - alone - into that environment. Now a whole programming language had to go in there.

That really ruled out about 90% of the programming languages that you could think of doing. So the question was, do we roll our own little, interpretive programming language, think of it as like mini-Perl, or something like that, or do we use an existing language. We were all, from the standpoint of compatibility, all for an existing language. And here was this thing called XLISP that was written by a fellow named David Betz who was, at the time, an employee of Digital Equipment Corporation in Massachusetts. And it was this tiny little PC LISP interpreter. Pure interpreter. And because he was an employee of Digital Equipment, their policy at the time was that, if you did any work on your own, you could either give it to them or you could put it in the public domain. He put it in the public domain. So that meant we were free to use it without any royalties or encumbrance.

Now XLISP was actually a different dialect of Lisp: it was more towards the MACLISP dialect - the MIT Lisp - than Common Lisp. It was missing a lot of very important things including, for example, floating point numbers. I don't think it had any trig functions. So there were just huge gaps. But the core of LISP is four functions - you can theoretically, given those four or five functions, write the whole of the rest of it in Lisp, if you have to. That functionality was there, and a good part of the rest of what you expect. And it fit in 64K! So here was something, here was a complete programming language that by a simple matter of programming we could extend to do anything we want. The architecture of it was such that there wasn't any problem putting in floating point arithmetic, it was just a matter of setting it out and doing it. And when we were done, we could actually converge upon Common Lisp, which was not only a respectable programming language, it was - in the mid-80s - the hot thing. That's where all the AI money was going in. "AI is the wave of the future, Lisp is the language of AI!" There was a certain marketing sizzle to choosing Lisp, as well.

Now it's interesting - and we didn't even know this, at the time, that's how isolated we were from the UNIX community, because, again, we went directly from mainframes to PCs, we skipped UNIX, in there - but around that time Richard Stallman was looking for a programming language for Emacs, and he ended up with Lisp also. And so Lisp ended up being the programming language of both the most widely used text editor on the UNIX environment and AutoCAD. And, I don't know, he wasn't faced with the memory restrictions we were - because he was working on VAXes, at the time - but we both came up with essentially the same answer, completely independent of one another.

Lisp is very much still in use today, of course.

Sure - if you want a little glue for something, absolutely. Even after we had ADS I did probably half to three-quarters of the little things I did with AutoCAD in AutoLISP. It's just easier - you don't have to compile a program.

And there's a swing back in that direction from companies such as Microsoft, who've got an explorative, interpretive interface to F#, for instance, and Functional Programming is definitely having an overall resurgence. Do you track programming language evolution?

From time to time. I've never really been one of these programming language either fanatics or groupies or trend-followers. I mean all of these languages were invented in the 1950s. There's nothing really new. It's just every decade there's another fad and something comes to the fore. And these things were invented in the 1950s because they were all good for various things, and none of them are good for everything - that was tried, that was PL/I. And let me tell you, that didn't work. And because of the universality of computation, anything you can do in one, you can do in all the other ones. Unless there's something deliberately omitted, or something like that.

So it's just a question of what's the best tool for the job. When someone talks to me about Lisp vs. functional programming, vs. Prolog, vs. declarative languages, to me that's just like asking "what's better - a hammer or a screwdriver?" Well, do you need to turn a screw or pound on something? I would say that other than fixing legacy code, I do about 95% of all my new software development in Perl. And Perl is about seventeen times slower than C. Who cares? You've got a 3GHz machine and the thing runs for a second! What's the difference whether it runs in a seventeenth of a second or a second? And I think that's one of the reasons that people are going back to the interpretive environments, because they are, in the most part, more friendly to the developer and certainly a lot easier to diagnose when something goes wrong in the field. And with the speed of machines we have and unlimited memory, the advantage that you had from native code just isn't there anymore.

Besides, who's running native code anymore? The x86 processor doesn't run the code you put in memory. It takes that x86 code and disassembles it into a very long instruction word architecture, which is the way the microprocessor really works. So even if you're writing in assembly language, that's not the code you're really running. So what's the difference between that and just-in-time byte-code compilation in a Java or C# implementation? That's just one more level.

I have a floating point benchmark that I update. Every time a new language comes out, I put it on there. And do you know what's the fastest language from my floating point benchmark, right now? Running on an x86 architecture, running a benchmark that does exactly the same thing and produces exactly the same answers. It does a ray-trace design of a refractor telescope. It does this 100,000 times, and you measure how long it takes. Well, it's Visual Basic .NET. Visual Basic, with its byte-code interpreter, is actually faster than C. And if C is 1, Java is 1.027. So, again, around 3% on a 3GHz machine - who's going to see that? Java has a problem with its startup time, because a typical Java program pulls in all kinds of stuff from the disk. And that's probably where it's losing the 3%, if I ran it longer then it might come close to approaching parity, but there are rules.

And none of these differences are big - the big differences are when you get to the Perls and Pythons and the dynamic languages. But even GNU Lisp is only seven times slower than C.

When Lisp was introduced, was the focus on user-level customization, professional development, or both?

I would say, really, both. I think what we had found was the developers - who prior to that, using menu macros, had done amazing things - had all started as users. They were simply users who found, "oh gee - look!" And, as I would describe it, they would always say "no, I'm not a programmer." What are they doing? They're programming - they just don't know it, because programming is this arcane thing that these people do at Microsoft, or something. But no - they're programming, and they just get sucked into the dark side, basically, by... first it's a menu macro, then it's a little variables and expressions, then it's AutoLISP... before you know it they're building big applications.

So there was that desire. We never even remotely considered making the development module something we would sell or license to official developers, or something like that. Everybody was going to have it, it was going to be in every product. Because we expected most of our developers were people who started out as users and built something interesting. We'd seen that hundreds of times. The whole UNIVAC community had a thriving, giveaway, third-party software market for things that UNIVAC mainframe users had developed for their own use, and said, "well, we're a government agency, we don't sell software, here it is."

And you might not know this, but DBase II, the Ashton-Tate database system, was a copy, a reimplementation of a package developed at the Jet Propulsion Laboratory called JPLDIS. In FORTRAN on UNIVAC mainframes. Everybody used JPLDIS. UNIVAC never expected Jet Propulsion Laboratory to develop a database system that everybody used. But suddenly you had people buying UNIVAC machines to run JPLDIS.

So there was that component, that we wanted to provide something to everybody, so our application developers would spontaneously appear, from users who found themselves creating applications. But also from the people - particularly Rik Jadrnicek and Dennis Neeley - who were responsible for what was first called AE/CADD and later became AutoCAD AEC Architectural, which was entirely a tablet of menu macros, at the epoch, we had a set of specific requirements for what they needed to go beyond what they had. And so those requirements went into what we put in. Not really so much for Variables & Expressions but the full AutoLISP. The ability, particularly, to reach in and look at the entities in the database and get the attributes and things.

Were they part of Autodesk?

No. Dennis Neeley was an architect, who continued to do architecture, Rik Jadrnicek was a consultant who Dennis Neeley brought in when he discovered it had gone beyond what he knew how to do. They had their own business relationship. Eventually, we licensed their product through a royalty deal, and these were two of the first - well, they were not the first add-ons to AutoCAD, but they were the first ones that anybody bought. And it was originally called AE/CADD, and then they made one called Mechanical - which was not for mechanical engineering but for the mechanical aspects of architecture.

Ah yes, that's right - there were AutoCAD AEC Architectural and AutoCAD AEC Mechanical...

Which confused everybody. But they were giving this - Rik, being the programmer - was telling us, "this is what I need, this is what I can't do with these bloody macros that we've got here. I need a way to get values out of entities, I need a way to tell how long a line is." So that's where it became obvious that we needed to let the thing - at least, in read-only fashion - reach into the database.

So you had external input guiding the evolution of the API-set.

Exactly, as with the feature-set of AutoCAD we had the users providing guidance there. One thing we always did, and this persisted, really, - and you may still do it - but certainly until Carol arrived, we'd always have line developers go out and do demos at trade-shows. Because you'd come back from trade-shows and have forty, fifty feature requests that were things that you would never have understood why it was important until the guy that needed it stood there and showed you on a copy of AutoCAD what it was that they couldn't do. And I just can't imagine a software company not doing that, because it's $100M market research for free. And it costs you a week - or three or four days - of your developer's time to get it.

And it was really the same thing. Once, particularly, with the guidance - because Rik had done far more than anybody else, and he was also in California, in the San Francisco area, so he was always at our office, pounding on the table and saying, "I need this" and "that works", believe me - but once AutoLISP was out there it was, again, the very same thing. Suddenly we got a flood of "well, I can't do this" and "you need to give me access to that" and "how do I turn this off". And so I've always found... people criticize Microsoft for getting some shoddy thing out there and improve it, but the advantage of getting the shoddy thing out there is that you start to get the feedback, of what it really needs to do as opposed to what you thought it needed to do.

I spent a year and a half developing this web application [The Hacker's Diet Online] and I put it out there, and I had a beta for a couple of months, and opened it up on the 2nd of July last year. And from Day 1 it had this little web feedback form, "bug"/"request for feature", and I think I'm over my hundredth "request for feature" implemented in the application today. Almost all of them were "darn, why didn't I think of that?"

What were the drivers behind the introduction of ADS [the AutoCAD Development System]?

I would say as large a motivator as any other was the fact that application developers would not make the investment in making large applications if they had to give away their source code. And with AutoLISP, when you sold the application you gave away your source code. We had an obfuscator - the Kelvinator - but there were de-Kelvinators out there, and so forth. And we had really reached the point, both that the AutoLISP applications were getting so big that there was actually a problem, because you couldn't really segment an AutoCAD application all the easily, I mean you could have things chain-load and so forth, but it was just maintaining code that big. And particularly if you're not doing things the Lisp way. I mean some of the very largest programs ever written have been written in Lisp, but they aren't written procedurally the way C people write things. What we had was a bunch of developers who were thinking BASIC or C and writing it in Lisp, and after a while it gets pretty unwieldy for that kind of thing. Obviously with ADS you got a huge performance benefit - not as much as you might think, if you're doing a lot of computation, obviously, it was enormous, things that were designing things parametrically, for example, for that kind of stuff it was of enormous benefit - but still, the interface between ADS and AutoCAD was the AutoLISP interface. ADS was essentially the code that AutoLISP used to talk to AutoCAD with an API put on the other end. Everything, at least initially, went through the same pipe that AutoLISP went through. So the only place that you got a real speed-up was on the pure C side, for computation. When it was talking to AutoCAD it didn't run any faster than AutoLISP did.

That changed, as it evolved. The biggest motivation - and the biggest reason for its adoption - was the ability to make binary applications that could not be reverse-engineered.

Were you involved in Release 13, at all?

R12 was the last release that I had any involvement with.

So you weren't involved in the decision to rearchitect the product during that release?

I was railing at the executive level for the fact that that needed to be done, and hearing that - this would have been like '91-'92 - we wouldn't have a fully-integrated Windows API version that looked like a Windows application until Release 15 - or what was called Release 15, at that point - which might be three or four years out. And I said, "that is death for the company". And they said, "well, it can't be done". And I said, "you have $160M in the bank - you can do a lot of things with $160M, if the future of the company is at stake." And I think eventually the message got home, that they did need to do a massive rearchitecture of the program. But that was really after management was changed and people who thought about what needs to be done, rather than what can be done, were in charge.

AutoCAD evolved with AutoLISP and ADS, and at some point we needed to be able to make modules that could be loaded directly inside AutoCAD that had the performance profile of DLLs, and also allowed us to extend AutoCAD without having direct source access, both internally and externally. That I always see as a significant decision. Although R13 was a difficult release, especially from a quality perspective, it needed to happen because AutoCAD would otherwise have died long ago.

It's always difficult... everybody makes this mistake, when building software, and they make it over and over and over again, serially for their whole careers. They start the program and they say "it's absolutely important that this application be completely open, that we expose everything that we do inside to the outside. That we create no barrier between our native code and the developer. That we give the developer every power to extend the program that we have. " And they go in saying that, and why do they go in saying that? Because they knew that the last program they worked on, they had to go and rearchitect the whole thing when they suddenly realized that they'd need it to be open.

And they start developing the thing, and it's open, and then, nobody's using that openness yet, and so it closes. And so you put this thing in, but you can't ship it in time, if this is all done outside or exposed. And eventually you get to that horrible realization that it's happened again, and that you have an architecture where there's a whole bunch of stuff going on inside there that you can't do from outside, and then you pay many times the cost that it would have been to do it right the first time and expose that.

I think if you'd sat down and talked with the people developing AutoCAD in 1982 and asked them, "how are you going to architect the program?" They would have all said, "it's got to be completely open." But, doggone it, first, you've got a 640K limit, second, you've got to get to market, and third, you've got users and dealers who are clamoring for features, they say "I could sell four times more", or "I would buy this if it had X", and you don't delay the product for three or four months to go and build it that way. And that's just the reality of software development. And I think that in the entire history of the software industry, there has not been a single, successful product that did not get used for things vastly larger and more complicated than its developers ever imagined it would be used for. And it's difficult to architect something for something that you can't imagine. That problem gets you, as well.

But again, I did a project in the late '80s, completely forgotten, except on my web-site, called ATLAST - we had to rename it from ATLAS because of a trademark problem. That was basically a Forth-based, of all things, API. So that you could build your program out of little chunks and glue them together, with the language. But the whole concept was - first of all, it ran as fast as native C code, so there was no hit at all in doing it this way as opposed to coding it in native C - but it meant that if you built your own application this way, you couldn't avoid exposing your entire functionality to developers. Because it was there, your own application used the same little bits with the glue code, as your developers had. From a strategic standpoint, if you didn't want to expose something, you could put a wall up, but it meant that technologically there was no barrier that separated you from the developers. Thank goodness we never did that, because Forth was not a memory-safe language.

The only thing that ATLAST ever got used for was a little project that I threw together in about a week, a week and a half, and it was just to get the goat of Jim Meadlock at Intergraph. I wanted to really irritate him. Intergraph had just launched this massive campaign - this was during the object-oriented fad - "Intergraph is an object-oriented CAD system." I can't imagine, really, how they justified it. I never really saw their argument. I think it was just like, I don't know, selling this 1950s Cadillac "with Nuclear Power!", or something like that. There wasn't anything object-oriented about it.

So I built this thing with ADS and ATLAST, that lets you have entities in AutoCAD that actually had methods. And you could send messages to them, and it had the big four of object-oriented: it had messages, it had inheritance, it had polymorphism... it was all there. And so I called this ClassWar, which stood for Class Language Application Support System Within AutoCAD, Really! I demoed this thing at one of our shows and said "this is an object-oriented CAD system!" And I never did get a reaction from Meadlock.

[Tune in next week for Part 3 - Autodesk's eco-system and strategy.]