Through the Interface

Batch-processing AutoCAD drawings from LISP without SDI (take 2)

In this recent post we looked at an approach combining AutoLISP with a script generated on-the-fly to get around the fact that (command "_.OPEN" …) does nothing when SDI == 0. As mentioned in a comment in the post, I realised that the approach of using a single master script to do this is more prone to failure: a number of commands can cause scripts to stop executing, for instance, so it would be better practice to minimise the operations contained in a particular script to increase the application’s fault tolerance.

This modified approach was suggested by a member of our Engineering team in a recent thread (one that I came across after Monday’s post). It uses a data file to store a list of the drawings to process and only creates a script to load – and launch processing on – the next drawing in that list:

(defun C:BATCH(/ dwgs lsp-name data-name)

  (setq dwgs '("C:/A.DWG" "C:/B.DWG" "C:/C.DWG" "C:/D.DWG")

        lsp-name "c:/tmp.lsp"

        data-name "c:/dwgs.tmp"

  )

  (create-drawing-list data-name dwgs)

  (process-next-drawing data-name lsp-name "(create-circle)" T T)

  (princ)

)

 

(defun create-circle()

  (command "_.CIRCLE" "10,10,0" "5")

)

 

(defun create-drawing-list(data dwgs / f dwg)

 

  ;; Create data file containing the DWG names

 

  (setq f (open data-name "w"))

  (foreach dwg dwgs

    (write-line dwg f)

  )

  (close f)

)

 

;; Get the first drawing from the list, removing it

 

(defun get-next-drawing(data / dwg dwgs f)

 

  ;; Read in the whole list of DWGs

 

  (setq f (open data "r")

        dwgs '()

  )

  (while (setq dwg (read-line f))

    (setq dwgs (cons dwg dwgs))

  )

  (close f)

 

  ;; Reverse the list, take the head and write

  ;; back the remainder to the file

 

  (if (> (length dwgs) 0)

    (progn

      (setq dwgs (reverse dwgs)

            dwg (car dwgs)

            dwgs (cdr dwgs)

      )

 

      (setq f (open data "w"))

      (foreach dwg dwgs

        (write-line dwg f)

      )

      (close f)

    )

  )

  dwg

)

 

;; Process the current drawing and use a script to open

;; the next one in the list

 

(defun process-next-drawing(data lsp func save first / scr)

  (setq scr "c:/tmp.scr")

 

  ;; We only want to run the function if not the first

  ;; time called... the same for save

 

  (if (not first)

    (progn

      (eval (read func))

      (if save

        (command "_.QSAVE")

      )

    )

  )

 

  ;; Get the next DWG name from the file

 

  (setq dwg (get-next-drawing data))

 

  ;; If there is one, create a script to open it, reload

  ;; the application and process our function

 

  (if dwg

    (progn

      (create-script scr data dwg lsp save first)

      (command "_.SCRIPT" scr)

    )

 

    ;; For the last drawing we simply close it and

    ;; delete the now-empty data file

 

    (progn

      (vl-file-delete data)

      (vl-file-delete scr)

      (command "_.CLOSE")

    )

  )

) 

 

;; Create a script to close the current drawing and

;; open the next, calling back to our process function

;; (after having loaded the file that defines it)

 

(defun create-script(scr data dwg lsp save first / f)

  (setq f (open scr "w"))

  (if (not first)

    (write-line "_.CLOSE" f)

  )

  (write-line

    (strcat "_.OPEN \"" dwg "\"") f

  )

  (write-line

    (strcat "(load \"" lsp "\")") f

  )

  (write-line

    (strcat

      "(process-next-drawing \""

      data "\" \"" lsp "\" \"" func "\" "

      (if save "T" "nil") " nil)"

    )

    f

  )

  (close f)

  (princ)

)

I hope this is useful to people – as mentioned before, please do provide feedback on how/whether this works for you…

Batch-processing AutoCAD drawings from LISP without SDI

As mentioned in this previous post, there has been some discussion internally around the future of SDI. Given the change this will bring to applications, SDI is going to be around until we next deliberately choose to break binary application compatibility (something we just did with AutoCAD 2010 and typically try to do only every three releases).

That said, SDI is very likely to go away at some point, so it does seem worth drilling further into the reasons for using it and trying to determine an appropriate way to remove current dependencies on it.

Thanks to all of you who responded to my previous post and provided input on your use of SDI. The most common theme was around the use of SDI to batch-process sets of drawings: opening each one, performing an operation and (optionally) saving before opening the next.

My understanding is that SDI makes life easier, in this situation, because the closing of one drawing is taken care of automatically when opening the next and so LISP applications can operate more easily across multiple drawings.

I did some tests, to see how it helps, but the LISP part of my brain has unfortunately atrophied, over the years: I wasn’t able to get a simple SDI-only, batch processing application to work, as it always stopped once a new drawing was open. It may be that some use of script files is needed – and this certainly can make life easier, as we’ll see below – but it would be good if someone could help me out by posting a comment or dropping me an email. I’m sure I’m missing something very simple.

Anyway, irrespective of whether I was able to use SDI successfully, or not, I was able to get something working from an MDI environment that I hope to be equivalent, functionality-wise.

Thanks to guidance from Wayne Brill, a member of DevTech Americas, I was able to put together some LISP code that makes use of a temporary script file to handle the opening, processing, (saving) and closing of drawings.

Here’s the LISP application:

(defun C:BATCH(/ dwgs scr-name lsp-name)

  (setq dwgs '("C:/A.DWG" "C:/B.DWG" "C:/C.DWG" "C:/D.DWG")

        scr-name "c:/tmp.scr"

        lsp-name "c:/batch.lsp"

  )

  (create-script scr-name dwgs lsp-name "(CreateCircle)" T)

  (command "_.SCRIPT" scr-name)

  (vl-file-delete scr-name)

  (princ)

)

 

(defun CreateCircle()

  (command "_.CIRCLE" "0,0,0" "30")

)

 

(defun create-script(scr dwgs lsp cmd save / f dwg)

  (setq f (open scr "w"))

  (foreach dwg dwgs

    (progn

      (write-line

        (strcat "_.OPEN \"" dwg "\"") f

      )

      (write-line

        (strcat "(load \"" lsp "\")") f

      )

      (write-line cmd f)

      (if save

        (write-line "_.QSAVE" f)

      )

      (write-line "_.CLOSE" f)

    )

  )

  (close f)

  (princ)

)

The script handles the opening of each drawing, reloading the LISP file (which I have saved in c:/tmp.lsp – this file is pointed at by the lsp-name variable in the C:BATCH function) inside each one and running the specified command/function before saving & closing. In this case we’re running a simple function that uses a command to create a circle – if we were doing something that didn’t require the drawing to be saved (if we were just querying data, for instance) we could pass nil instead of T into the (create-script) function.

To take a look at the script being used behind the scenes, you can simply comment out the call to (vl-file-delete) and open up the contents in your favourite text editor:

_.OPEN "C:/A.DWG"

(load "c:/tmp.lsp")

(CreateCircle)

_.QSAVE

_.CLOSE

_.OPEN "C:/B.DWG"

(load "c:/tmp.lsp")

(CreateCircle)

_.QSAVE

_.CLOSE

_.OPEN "C:/C.DWG"

(load "c:/tmp.lsp")

(CreateCircle)

_.QSAVE

_.CLOSE

_.OPEN "C:/D.DWG"

(load "c:/tmp.lsp")

(CreateCircle)

_.QSAVE

_.CLOSE

I hope this approach goes some way towards helping people batch-process drawings from LISP without having to move to a different language. I’d really appreciate your feedback on this subject, as it would be good to get a more definitive approach nailed down before this change (eventually) becomes a requirement.

Using IronPython with AutoCAD

I’ve been meaning to play around with the Python language for some time, now, and with the recent release of IronPython 2 it seems a good time to start.

Why Python? A number of people in my team – including Jeremy Tammik and the people within our Media  & Entertainment workgroup who support Python’s use with Maya and MotionBuilder – are fierce proponents of the language. I’m told that it’s an extremely easy, general-purpose, dynamic programming language. All of which sounds interesting, of course, although I have to admit I’m less convinced of the importance of the dynamic piece: I’ve found a lot of value in static typing over the years (even F# is statically typed, although many people – even some who work with it - don’t realise this… its type inference system allows you to code safely without specifying types all over the place).

Let’s take a quick step back and talk about what makes a language dynamic. The most common example of a dynamic language – one that I’m sure most of you will have touched at some point – is JavaScript. In JavaScript you declare everything as a var, assign it, call methods on it and hope that they work at runtime. I admit that I’ve always disliked developing in JavaScript because of the lack of decent tool support: I’m a big fan of Intellisense (based on an object’s design-time type) and want the compiler to tell me if I’m dealing with an object that doesn’t support a particular method. But perhaps that’s largely what I’ve become used to from modern development tools, and I’m trying to remain open to new things. Really, I am.

Another dynamic language with which I’ve had much more favourable (but still, at times, frustrating) experiences is LISP. But my relationship with LISP is different: like most early AutoCAD programmers I adopted it out of necessity – and at the time I started with it programming environments were, in any case, generally very basic - I then grew to love it and have since never forgotten it, even when more attractive/productive development environments came along. So I’m extremely loathe to paint it with the same brush as the one I’ve used for JavaScript.

Python is also of interest because of its cross-platform availability: it’s an open source language with its roots in the UNIX/Linux world, but is now gaining popularity across a variety of OS platforms (one of the reasons it’s the scripting language chosen for at least one of our cross-platform products, Autodesk Maya).

Microsoft is definitely now very open to the possibilities of dynamic languages: they’re making a significant investment in the Dynamic Language Runtime, to support languages such as IronPython and IronRuby, as well as adding more dynamic features with C# 4.0 (which is finally going to get something comparable to VB’s “late binding” capability).

So all in all, the world we live in seems to be becoming increasingly dynamic. :-)

Anyway – now on to getting IronPython working with AutoCAD. I had originally hoped to build a .NET assembly directly using IronPython – something that appears to have been enabled with the 2.0 release of IronPython - which could then be loaded into AutoCAD. Unfortunately this was an exercise in frustration: AutoCAD makes heavy use of custom attributes for identifying commands etc., but IronPython doesn’t currently support the use of attributes. It is possible to do some clever stuff by compiling attributed C# on-the-fly and deriving classes from it (information on this is available here), which will – in theory, at least – get you something in memory that’s attributed but, as AutoCAD scans the physical assembly for custom attributes before loading it, this didn’t help. I also spent a great deal of time just trying to derive a class from Autodesk.AutoCAD.Runtime.IExtensionApplication – to have the Initialize() function called automatically on load – but I just couldn’t get this to work, either.

Then, thankfully, Tim Riley came to the rescue: we’ve been in touch on and off over the years since he started the PyAcad.NET project to run IronPython code inside AutoCAD, and Tim was able to put together some working code which actually registered commands (after I’d pointed him at a function he could use from AutoCAD 2009’s acmgdinternal.dll – an unsupported assembly that exposes some otherwise quite helpful functions). He ended up choosing an implementation that had also been suggested to me by Albert Szilvasy: to implement a PYLOAD command using C# which allows selection and loading of a Python script (because Python is, ultimately, all about scripting rather than building static, compiled assemblies).

Before we get on to the C# module, I should point out that I installed IronPython 2.0.1 as well as IronPython Studio 1.0 for the Visual Studio 2008 integration. It turns out that as we’re relying on C# to manage the loading of Python – rather than compiling a .NET assembly – the main advantage of IronPython Studio is around the ability to work with Python source code inside Visual Studio.

To build the below C# code into a standard .NET Class Library assembly (a .DLL) you’ll need to add assembly references to IronPython.dll, IronPython.Modules.dll, Microsoft.Scripting.dll and Microsoft.Scripting.Core.dll – all of which can be found in the main IronPython install folder (on my system this is in “C:\Program Files\IronPython 2.0.1”). As well as the standard references to acmgd.dll and acdbmgd.dll, of course.

Here’s the C# code:

using Autodesk.AutoCAD.ApplicationServices;

using Autodesk.AutoCAD.DatabaseServices;

using Autodesk.AutoCAD.Runtime;

using Autodesk.AutoCAD.EditorInput;

using IronPython.Hosting;

using Microsoft.Scripting.Hosting;

using System;

 

namespace PythonLoader

{

  public class CommandsAndFunctions

  {

    [CommandMethod("-PYLOAD")]

    public static void PythonLoadCmdLine()

    {

      PythonLoad(true);

    }

 

    [CommandMethod("PYLOAD")]

    public static void PythonLoadUI()

    {

      PythonLoad(false);

    }

 

    public static void PythonLoad(bool useCmdLine)

    {

      Document doc =

        Application.DocumentManager.MdiActiveDocument;

      Editor ed = doc.Editor;

 

      short fd =

        (short)Application.GetSystemVariable("FILEDIA");

 

      // As the user to select a .py file

 

      PromptOpenFileOptions pfo =

          new PromptOpenFileOptions(

            "Select Python script to load"

          );

      pfo.Filter = "Python script (*.py)|*.py";

      pfo.PreferCommandLine =

        (useCmdLine || fd == 0);

      PromptFileNameResult pr =

        ed.GetFileNameForOpen(pfo);

 

      // And then try to load and execute it

 

      if (pr.Status == PromptStatus.OK)

        ExecutePythonScript(pr.StringResult);

    }

 

    [LispFunction("PYLOAD")]

    public ResultBuffer PythonLoadLISP(ResultBuffer rb)

    {

      const int RTSTR = 5005;

 

      Document doc =

        Application.DocumentManager.MdiActiveDocument;

      Editor ed = doc.Editor;

 

      if (rb == null)

      {

        ed.WriteMessage("\nError: too few arguments\n");

      }

      else

      {

        // We're only really interested in the first argument

 

        Array args = rb.AsArray();

        TypedValue tv = (TypedValue)args.GetValue(0);

 

        // Which should be the filename of our script

 

        if (tv != null && tv.TypeCode == RTSTR)

        {

          // If we manage to execute it, let's return the

          // filename as the result of the function

          // (just as (arxload) does)

 

          bool success =

            ExecutePythonScript(Convert.ToString(tv.Value));

          return

            (success ?

              new ResultBuffer(

                new TypedValue(RTSTR, tv.Value)

              )

              : null);

        }

      }

      return null;

    }

 

    private static bool ExecutePythonScript(string file)

    {

      // If the file exists, let's load and execute it

      // (we could/should probably add some more robust

      // exception handling here)

 

      bool ret = System.IO.File.Exists(file);

      if (ret)

      {

        ScriptEngine engine = Python.CreateEngine();

        engine.ExecuteFile(file);

      }

      return ret;

    }

  }

}

The code behind the PYLOAD command is actually really simple. I could have kept it basic but decided it would be a good opportunity to show some best practices. So not only do we have the standard PYLOAD command, which respects the FILEDIA variable to decide whether to use dialogs or the command-line, we also have a command-line version –PYLOAD and a LISP function (pyload). All of which call into the same function to load a Python script.

OK, now let’s take a look at a simple IronPython script that calls into AutoCAD via its .NET API. Thanks again to Tim Riley for providing something that works. Even with Python being (apparently) so easy to learn, I’m such a neophyte that without his help I’d still be stumbling around in the dark.

import clr

path = 'C:\\Program Files\\Autodesk\\AutoCAD 2009\\'

clr.AddReferenceToFileAndPath(path + 'acdbmgd.dll')

clr.AddReferenceToFileAndPath(path + 'acmgd.dll')

clr.AddReferenceToFileAndPath(path + 'acmgdinternal.dll')

 

import Autodesk

import Autodesk.AutoCAD.Runtime as ar

import Autodesk.AutoCAD.ApplicationServices as aas

import Autodesk.AutoCAD.DatabaseServices as ads

import Autodesk.AutoCAD.Geometry as ag

import Autodesk.AutoCAD.Internal as ai

from Autodesk.AutoCAD.Internal import Utils

 

# Function to register AutoCAD commands

# To be used via a function decorator

 

def autocad_command(function):

 

    # First query the function name

    n = function.__name__

 

    # Create the callback and add the command

    cc = ai.CommandCallback(function)

    Utils.AddCommand('pycmds', n, n, ar.CommandFlags.Modal, cc)

 

    # Let's now write a message to the command-line

    doc = aas.Application.DocumentManager.MdiActiveDocument

    ed = doc.Editor

    ed.WriteMessage("\nRegistered Python command: {0}", n)

 

# A simple "Hello World!" command

 

@autocad_command

def msg():

    doc = aas.Application.DocumentManager.MdiActiveDocument

    ed = doc.Editor

    ed.WriteMessage("\nOur test command works!")

 

# And one to do something a little more complex...

# Adds a circle to the current space

 

@autocad_command

def mycir():

 

    doc = aas.Application.DocumentManager.MdiActiveDocument

    db = doc.Database

 

    tr = doc.TransactionManager.StartTransaction()

    bt = tr.GetObject(db.BlockTableId, ads.OpenMode.ForRead)

    btr = tr.GetObject(db.CurrentSpaceId, ads.OpenMode.ForWrite)

 

    cir = ads.Circle(ag.Point3d(10,10,0),ag.Vector3d.ZAxis, 2)

 

    btr.AppendEntity(cir)

    tr.AddNewlyCreatedDBObject(cir, True)

 

    tr.Commit()

    tr.Dispose()

As we’re stuck without the ability to use custom attributes in IronPython, we’re making use of the Autodesk.AutoCAD.Internal namespace to register commands at runtime. I don’t like doing this, but at the same time I was left with little choice, unless we choose to find another way to call into the code. Please be warned that anything contained in the Autodesk.AutoCAD.Internal namespace is unsupported functionality, and subject to change without warning.

Now that I have that off my chest, let’s comment a little further on the above code…

• Even without custom attributes, we have used a pretty cool Python language feature known as decorators (thanks *again* for the tip, Tim :-) which helps us to mark functions as commands. The autocad_command function is called for each decorated function, and this is where we register a command for the function based on the function’s name. Pretty cool.
• You’ll notice a distinct lack of types in the code (and yes, that still scares me). When I was previously trying to compile a DLL based on this code, I had a lot of trouble getting anything at all to fail at compile-time, but clearly a lot would fail at runtime (when I could actually get anything to execute :-S). I feel as though I still need to get my head around this trade-off: I can see the argument for simplicity/elegance/succinctness – and even the power it brings in some situations - but the Computer Scientist in me is screaming for safety/reliability/determinism/debuggability (if that’s even a word). Oh well. The main thing is that I’m starting the journey, at least: we’ll see if it ends up somewhere I like. :-)

When we build and NETLOAD our PythonLoader C# application and execute the PYLOAD command, we can select our Python script:

Command: PYLOAD

Once selected, the script gets loaded and should register a couple of commands:

Registered Python command: msg

Registered Python command: mycir

Running the MSG command will execute a simple “Hello World!”-like function, just printing a message to the command-line:

Command: MSG

Our test command works!

And running the MYCIR command should just add a simple circle to the current space in the active drawing.

Command: MYCIR

That’s it for my initial foray into the world of Python. I hope you’ve found this helpful and enjoy playing around with the Python programming language inside AutoCAD. Please do post a comment if you have experiences or anecdotes to share on this topic!

Free AutoCAD programming webcast: .NET for LISP Programmers

As mentioned in this previous post, my team delivers regular webcasts on programming with Autodesk technologies.

Assuming you're reading this while the metaphorical ink's still damp, it's still not too late to register for a class being held on Thursday October 16th at 8am Pacific, 5pm CET: AutoCAD: .NET for LISP Programmers.

Otherwise you should be able to find a recording of the webcast on this page within a few weeks of the session being delivered. In fact, here's the recording of the last time this session was delivered (although it may well have been updated slightly, as the session will also be delivered at AU 2008).

Here's a brief synopsis of the topic covered by this webcast:

You know that .NET is a modern and powerful programming environment. But, no matter how much you might want to start using it, you can't abandon all your old LISP code. We’ll show you how to write .NET code that can interoperate with your existing LISP code, with a special emphasis on user interface components.

An interview with John Walker - Part 2

This is the second part of the series documenting my recent interview with John Walker. Where the first part looked at the early history of the company, this part focuses more on the architectural evolution of AutoCAD and its APIs. I have not linked to terms previously linked to in the first part of the series, which should also be referred to for typographical conventions, if they're not clear.

• Part 1 - Autodesk's early history
• Part 2 - AutoCAD's architecture & APIs
• Part 3 - Autodesk's eco-system and strategy
• Part 4 - Past and future opportunities

AutoCAD's architecture & APIs

We've talked a little about Variables and Expressions, already. What was behind the decision to use Lisp as a language?

Well, that's interesting. I would say, as much as anything, opportunity. We had this menu macro language, and that clearly had gone as far as it could go. We needed something that would provide access to the database. We needed something that could actually do algorithmic operations needed in writing the program. And it had to fit in 64K. It had to fit in a 64K segment, basically, and it had to fit - with that 64K segment - in a 640K IBM PC environment. Along with all the rest of AutoCAD, which at that point was hideously overlaid just to squeeze AutoCAD - alone - into that environment. Now a whole programming language had to go in there.

That really ruled out about 90% of the programming languages that you could think of doing. So the question was, do we roll our own little, interpretive programming language, think of it as like mini-Perl, or something like that, or do we use an existing language. We were all, from the standpoint of compatibility, all for an existing language. And here was this thing called XLISP that was written by a fellow named David Betz who was, at the time, an employee of Digital Equipment Corporation in Massachusetts. And it was this tiny little PC LISP interpreter. Pure interpreter. And because he was an employee of Digital Equipment, their policy at the time was that, if you did any work on your own, you could either give it to them or you could put it in the public domain. He put it in the public domain. So that meant we were free to use it without any royalties or encumbrance.

Now XLISP was actually a different dialect of Lisp: it was more towards the MACLISP dialect - the MIT Lisp - than Common Lisp. It was missing a lot of very important things including, for example, floating point numbers. I don't think it had any trig functions. So there were just huge gaps. But the core of LISP is four functions - you can theoretically, given those four or five functions, write the whole of the rest of it in Lisp, if you have to. That functionality was there, and a good part of the rest of what you expect. And it fit in 64K! So here was something, here was a complete programming language that by a simple matter of programming we could extend to do anything we want. The architecture of it was such that there wasn't any problem putting in floating point arithmetic, it was just a matter of setting it out and doing it. And when we were done, we could actually converge upon Common Lisp, which was not only a respectable programming language, it was - in the mid-80s - the hot thing. That's where all the AI money was going in. "AI is the wave of the future, Lisp is the language of AI!" There was a certain marketing sizzle to choosing Lisp, as well.

Now it's interesting - and we didn't even know this, at the time, that's how isolated we were from the UNIX community, because, again, we went directly from mainframes to PCs, we skipped UNIX, in there - but around that time Richard Stallman was looking for a programming language for Emacs, and he ended up with Lisp also. And so Lisp ended up being the programming language of both the most widely used text editor on the UNIX environment and AutoCAD. And, I don't know, he wasn't faced with the memory restrictions we were - because he was working on VAXes, at the time - but we both came up with essentially the same answer, completely independent of one another.

Lisp is very much still in use today, of course.

Sure - if you want a little glue for something, absolutely. Even after we had ADS I did probably half to three-quarters of the little things I did with AutoCAD in AutoLISP. It's just easier - you don't have to compile a program.

And there's a swing back in that direction from companies such as Microsoft, who've got an explorative, interpretive interface to F#, for instance, and Functional Programming is definitely having an overall resurgence. Do you track programming language evolution?

From time to time. I've never really been one of these programming language either fanatics or groupies or trend-followers. I mean all of these languages were invented in the 1950s. There's nothing really new. It's just every decade there's another fad and something comes to the fore. And these things were invented in the 1950s because they were all good for various things, and none of them are good for everything - that was tried, that was PL/I. And let me tell you, that didn't work. And because of the universality of computation, anything you can do in one, you can do in all the other ones. Unless there's something deliberately omitted, or something like that.

So it's just a question of what's the best tool for the job. When someone talks to me about Lisp vs. functional programming, vs. Prolog, vs. declarative languages, to me that's just like asking "what's better - a hammer or a screwdriver?" Well, do you need to turn a screw or pound on something? I would say that other than fixing legacy code, I do about 95% of all my new software development in Perl. And Perl is about seventeen times slower than C. Who cares? You've got a 3GHz machine and the thing runs for a second! What's the difference whether it runs in a seventeenth of a second or a second? And I think that's one of the reasons that people are going back to the interpretive environments, because they are, in the most part, more friendly to the developer and certainly a lot easier to diagnose when something goes wrong in the field. And with the speed of machines we have and unlimited memory, the advantage that you had from native code just isn't there anymore.

Besides, who's running native code anymore? The x86 processor doesn't run the code you put in memory. It takes that x86 code and disassembles it into a very long instruction word architecture, which is the way the microprocessor really works. So even if you're writing in assembly language, that's not the code you're really running. So what's the difference between that and just-in-time byte-code compilation in a Java or C# implementation? That's just one more level.

I have a floating point benchmark that I update. Every time a new language comes out, I put it on there. And do you know what's the fastest language from my floating point benchmark, right now? Running on an x86 architecture, running a benchmark that does exactly the same thing and produces exactly the same answers. It does a ray-trace design of a refractor telescope. It does this 100,000 times, and you measure how long it takes. Well, it's Visual Basic .NET. Visual Basic, with its byte-code interpreter, is actually faster than C. And if C is 1, Java is 1.027. So, again, around 3% on a 3GHz machine - who's going to see that? Java has a problem with its startup time, because a typical Java program pulls in all kinds of stuff from the disk. And that's probably where it's losing the 3%, if I ran it longer then it might come close to approaching parity, but there are rules.

And none of these differences are big - the big differences are when you get to the Perls and Pythons and the dynamic languages. But even GNU Lisp is only seven times slower than C.

When Lisp was introduced, was the focus on user-level customization, professional development, or both?

I would say, really, both. I think what we had found was the developers - who prior to that, using menu macros, had done amazing things - had all started as users. They were simply users who found, "oh gee - look!" And, as I would describe it, they would always say "no, I'm not a programmer." What are they doing? They're programming - they just don't know it, because programming is this arcane thing that these people do at Microsoft, or something. But no - they're programming, and they just get sucked into the dark side, basically, by... first it's a menu macro, then it's a little variables and expressions, then it's AutoLISP... before you know it they're building big applications.

So there was that desire. We never even remotely considered making the development module something we would sell or license to official developers, or something like that. Everybody was going to have it, it was going to be in every product. Because we expected most of our developers were people who started out as users and built something interesting. We'd seen that hundreds of times. The whole UNIVAC community had a thriving, giveaway, third-party software market for things that UNIVAC mainframe users had developed for their own use, and said, "well, we're a government agency, we don't sell software, here it is."

And you might not know this, but DBase II, the Ashton-Tate database system, was a copy, a reimplementation of a package developed at the Jet Propulsion Laboratory called JPLDIS. In FORTRAN on UNIVAC mainframes. Everybody used JPLDIS. UNIVAC never expected Jet Propulsion Laboratory to develop a database system that everybody used. But suddenly you had people buying UNIVAC machines to run JPLDIS.

So there was that component, that we wanted to provide something to everybody, so our application developers would spontaneously appear, from users who found themselves creating applications. But also from the people - particularly Rik Jadrnicek and Dennis Neeley - who were responsible for what was first called AE/CADD and later became AutoCAD AEC Architectural, which was entirely a tablet of menu macros, at the epoch, we had a set of specific requirements for what they needed to go beyond what they had. And so those requirements went into what we put in. Not really so much for Variables & Expressions but the full AutoLISP. The ability, particularly, to reach in and look at the entities in the database and get the attributes and things.

Were they part of Autodesk?

No. Dennis Neeley was an architect, who continued to do architecture, Rik Jadrnicek was a consultant who Dennis Neeley brought in when he discovered it had gone beyond what he knew how to do. They had their own business relationship. Eventually, we licensed their product through a royalty deal, and these were two of the first - well, they were not the first add-ons to AutoCAD, but they were the first ones that anybody bought. And it was originally called AE/CADD, and then they made one called Mechanical - which was not for mechanical engineering but for the mechanical aspects of architecture.

Ah yes, that's right - there were AutoCAD AEC Architectural and AutoCAD AEC Mechanical...

Which confused everybody. But they were giving this - Rik, being the programmer - was telling us, "this is what I need, this is what I can't do with these bloody macros that we've got here. I need a way to get values out of entities, I need a way to tell how long a line is." So that's where it became obvious that we needed to let the thing - at least, in read-only fashion - reach into the database.

So you had external input guiding the evolution of the API-set.

Exactly, as with the feature-set of AutoCAD we had the users providing guidance there. One thing we always did, and this persisted, really, - and you may still do it - but certainly until Carol arrived, we'd always have line developers go out and do demos at trade-shows. Because you'd come back from trade-shows and have forty, fifty feature requests that were things that you would never have understood why it was important until the guy that needed it stood there and showed you on a copy of AutoCAD what it was that they couldn't do. And I just can't imagine a software company not doing that, because it's $100M market research for free. And it costs you a week - or three or four days - of your developer's time to get it.

And it was really the same thing. Once, particularly, with the guidance - because Rik had done far more than anybody else, and he was also in California, in the San Francisco area, so he was always at our office, pounding on the table and saying, "I need this" and "that works", believe me - but once AutoLISP was out there it was, again, the very same thing. Suddenly we got a flood of "well, I can't do this" and "you need to give me access to that" and "how do I turn this off". And so I've always found... people criticize Microsoft for getting some shoddy thing out there and improve it, but the advantage of getting the shoddy thing out there is that you start to get the feedback, of what it really needs to do as opposed to what you thought it needed to do.

I spent a year and a half developing this web application [The Hacker's Diet Online] and I put it out there, and I had a beta for a couple of months, and opened it up on the 2nd of July last year. And from Day 1 it had this little web feedback form, "bug"/"request for feature", and I think I'm over my hundredth "request for feature" implemented in the application today. Almost all of them were "darn, why didn't I think of that?"

What were the drivers behind the introduction of ADS [the AutoCAD Development System]?

I would say as large a motivator as any other was the fact that application developers would not make the investment in making large applications if they had to give away their source code. And with AutoLISP, when you sold the application you gave away your source code. We had an obfuscator - the Kelvinator - but there were de-Kelvinators out there, and so forth. And we had really reached the point, both that the AutoLISP applications were getting so big that there was actually a problem, because you couldn't really segment an AutoCAD application all the easily, I mean you could have things chain-load and so forth, but it was just maintaining code that big. And particularly if you're not doing things the Lisp way. I mean some of the very largest programs ever written have been written in Lisp, but they aren't written procedurally the way C people write things. What we had was a bunch of developers who were thinking BASIC or C and writing it in Lisp, and after a while it gets pretty unwieldy for that kind of thing. Obviously with ADS you got a huge performance benefit - not as much as you might think, if you're doing a lot of computation, obviously, it was enormous, things that were designing things parametrically, for example, for that kind of stuff it was of enormous benefit - but still, the interface between ADS and AutoCAD was the AutoLISP interface. ADS was essentially the code that AutoLISP used to talk to AutoCAD with an API put on the other end. Everything, at least initially, went through the same pipe that AutoLISP went through. So the only place that you got a real speed-up was on the pure C side, for computation. When it was talking to AutoCAD it didn't run any faster than AutoLISP did.

That changed, as it evolved. The biggest motivation - and the biggest reason for its adoption - was the ability to make binary applications that could not be reverse-engineered.

Were you involved in Release 13, at all?

R12 was the last release that I had any involvement with.

So you weren't involved in the decision to rearchitect the product during that release?

I was railing at the executive level for the fact that that needed to be done, and hearing that - this would have been like '91-'92 - we wouldn't have a fully-integrated Windows API version that looked like a Windows application until Release 15 - or what was called Release 15, at that point - which might be three or four years out. And I said, "that is death for the company". And they said, "well, it can't be done". And I said, "you have $160M in the bank - you can do a lot of things with $160M, if the future of the company is at stake." And I think eventually the message got home, that they did need to do a massive rearchitecture of the program. But that was really after management was changed and people who thought about what needs to be done, rather than what can be done, were in charge.

AutoCAD evolved with AutoLISP and ADS, and at some point we needed to be able to make modules that could be loaded directly inside AutoCAD that had the performance profile of DLLs, and also allowed us to extend AutoCAD without having direct source access, both internally and externally. That I always see as a significant decision. Although R13 was a difficult release, especially from a quality perspective, it needed to happen because AutoCAD would otherwise have died long ago.

It's always difficult... everybody makes this mistake, when building software, and they make it over and over and over again, serially for their whole careers. They start the program and they say "it's absolutely important that this application be completely open, that we expose everything that we do inside to the outside. That we create no barrier between our native code and the developer. That we give the developer every power to extend the program that we have. " And they go in saying that, and why do they go in saying that? Because they knew that the last program they worked on, they had to go and rearchitect the whole thing when they suddenly realized that they'd need it to be open.

And they start developing the thing, and it's open, and then, nobody's using that openness yet, and so it closes. And so you put this thing in, but you can't ship it in time, if this is all done outside or exposed. And eventually you get to that horrible realization that it's happened again, and that you have an architecture where there's a whole bunch of stuff going on inside there that you can't do from outside, and then you pay many times the cost that it would have been to do it right the first time and expose that.

I think if you'd sat down and talked with the people developing AutoCAD in 1982 and asked them, "how are you going to architect the program?" They would have all said, "it's got to be completely open." But, doggone it, first, you've got a 640K limit, second, you've got to get to market, and third, you've got users and dealers who are clamoring for features, they say "I could sell four times more", or "I would buy this if it had X", and you don't delay the product for three or four months to go and build it that way. And that's just the reality of software development. And I think that in the entire history of the software industry, there has not been a single, successful product that did not get used for things vastly larger and more complicated than its developers ever imagined it would be used for. And it's difficult to architect something for something that you can't imagine. That problem gets you, as well.

But again, I did a project in the late '80s, completely forgotten, except on my web-site, called ATLAST - we had to rename it from ATLAS because of a trademark problem. That was basically a Forth-based, of all things, API. So that you could build your program out of little chunks and glue them together, with the language. But the whole concept was - first of all, it ran as fast as native C code, so there was no hit at all in doing it this way as opposed to coding it in native C - but it meant that if you built your own application this way, you couldn't avoid exposing your entire functionality to developers. Because it was there, your own application used the same little bits with the glue code, as your developers had. From a strategic standpoint, if you didn't want to expose something, you could put a wall up, but it meant that technologically there was no barrier that separated you from the developers. Thank goodness we never did that, because Forth was not a memory-safe language.

The only thing that ATLAST ever got used for was a little project that I threw together in about a week, a week and a half, and it was just to get the goat of Jim Meadlock at Intergraph. I wanted to really irritate him. Intergraph had just launched this massive campaign - this was during the object-oriented fad - "Intergraph is an object-oriented CAD system." I can't imagine, really, how they justified it. I never really saw their argument. I think it was just like, I don't know, selling this 1950s Cadillac "with Nuclear Power!", or something like that. There wasn't anything object-oriented about it.

So I built this thing with ADS and ATLAST, that lets you have entities in AutoCAD that actually had methods. And you could send messages to them, and it had the big four of object-oriented: it had messages, it had inheritance, it had polymorphism... it was all there. And so I called this ClassWar, which stood for Class Language Application Support System Within AutoCAD, Really! I demoed this thing at one of our shows and said "this is an object-oriented CAD system!" And I never did get a reaction from Meadlock.

[Tune in next week for Part 3 - Autodesk's eco-system and strategy.]

AutoCAD 2009 Update 1 now available

The first update to AutoCAD 2009 is now available. Here is the Readme for this update, which includes a note that these developer-oriented fixes are included (as well as many more general issues):

.NET API

• When a .NET function with a LispFunction attribute has the return type as object, an exception is thrown in AutoCAD.

Visual LISP

• The AutoLISP Redraw function fails to hide or highlight objects.

Metaprogramming with AutoCAD - Part 1

A recent comment on one of my F# articles got me thinking about this topic (thanks, Thomas! :-), so I thought I’d write a few posts on it. Next week is AU, and the week after that I’m attending a training class in Boston, so posts may be a little sparse over the coming weeks.

Metaprogramming  – according to the definition on Wikipedia – is the act of writing code that writes or manipulates other programs (or itself). But what is it really all about? The vast majority of programmers are actually metaprogramming without realizing it has such a fancy name.

To help understand metaprogramming, we’re going to focus on two ways of categorizing the various types of metaprogramming activity. Metaprogramming is usually either static or dynamic and homogeneous or heterogeneous (there are other classifications, but we’re not going to worry about those in this article).

• Static = at compile-time
• Dynamic = at runtime
• Homogeneous = the same output language is used as on the input
• Heterogeneous = a different output language is used to the input language

The most obvious form of metaprogramming is to create machine-code using a compiler (or even an interpreter) for a high-level language. This is a static, heterogeneous act of metaprogramming (although using an interpreter would presumably make this dynamic). Here are a few more interesting examples of metaprogramming I’ve used myself:

• C++ templates or pre-processor macros to generate lower-level code at compile-time
  • Static (compile-time) and heterogeneous (the generation language is different from the output language)
• Generation of a series of LISP expressions that are evaluated at runtime
  • Dynamic (runtime) and homogeneous (the generation and output languages are the same)
• Programmatic creation (perhaps using LISP, C#, VB(A) or C++) of an AutoCAD script that is then executed
  • Dynamic and heterogeneous
• Composing a SQL statement on-the-fly and using it to query a database
  • Dynamic and heterogeneous

The focus of this series of posts is on dynamic metaprogramming, which allows the modification of code at runtime (rather than compile-time). A further, somewhat more complex, example of dynamic metaprogramming is to redefine functions at runtime (something that’s possible with LISP, for instance), which allows applications to evolve, such as when developing expert systems that “learn” over time.

LISP was really one of the early programming environments that enabled metaprogramming, primarily through its ability to evaluate expression using (eval) and to redefine functions at runtime using (defun). This has been immensely valuable to AutoLISP programmers over the years. When AutoLISP was first introduced it was purely an interpreted language, so dynamic metaprogramming was provided pretty much automatically. In order for metaprogramming to work after the introduction of Visual LISP (which is fundamentally a compiled environment, albeit to an intermediate language), a runtime component supporting dynamic compilation was needed and provided. Very little change was needed in AutoLISP code, although in some rare cases (defun-q) now needs to be used, if it’s important to provide access to the internal representation of functions.

We’ll see this is a common thread for dynamic metaprogramming: by definition you either need to be working in an interpreted environment or will need to have a runtime component available that supports some kind of compilation (probably JIT). Visual LISP provides this, as does VBA and .NET (via the CLR).

Back to AutoLISP: one very common activity is to interpret a string using (read) and then call (eval) on it. The string may have been stored in a drawing, a text file, an external database, or generated on-the-fly. For example:

Command: (eval (read "(* 5 (getvar \"ZOOMFACTOR\"))"))

300

VBA also has native support for dynamic metaprogramming via the Eval() function:

Eval "MsgBox ThisDrawing.Name"

VB6 doesn't have direct support for Eval(), but it seems you can make use of it either by embedding a Script Control or by calling across to the VBA runtime (Googling "VB6 Eval" returned a number of options). I don't know whether it's possible to evaluate and make use of AutoCAD-specific variables - such as ThisDrawing - when using these techniques, however.

Metaprogramming with .NET is not quite so automatic, but is altogether possible, as I’ll show in my next post.

Free webcast: "AutoCAD: .NET for LISP Programmers"

On October 18th we're delivering a free AutoCAD development-oriented webcast entitled "AutoCAD: .NET for LISP Programmers".

Check here for our overall API schedule, or go here to register directly. These sessions are typically held at the end of the working day in Europe (5pm CEST) which is the beginning of the day on the West Coast of the US (8am PDT).

As usual, I'll post a link to the recording, once it's available.

Loading the right version of an ObjectARX module into 32- or 64-bit AutoCAD

This question has come in from a number of developers...

How can I tell when my application is running inside a 64-bit version of AutoCAD?

As mentioned in this previous post, AutoCAD 2008 installs as native 64-bit binaries on a supported 64-bit OS, just as 32-bit binaries get installed on a supported 32-bit OS.

A minor complication is that certain of our AutoCAD-based products do not yet have native 64-bit versions. Our Engineering teams are working on this, but in the meantime, your application might well be working inside a 32-bit Autodesk product on a 64-bit OS.

So how do we know whether we're on a 32- or 64-bit platform (i.e. AutoCAD)?

The ideal would be to have a simple system variable (just like we have ACADVER for the version of AutoCAD), which can be queried from any environment. Unfortunately this has not (as yet) been provided, so we have to look for another approach, for now.

The good news is that .NET applications generally shouldn't care - the same binary will work on both 32- and 64-bit platforms. The same for LISP, but then people often use LISP loaders to load ObjectARX modules - which most certainly do care - so my feeling is that this problem will most commonly be faced from LISP.

Before talking about getting the information from LISP, let's talk a little about ObjectARX, first. ObjectARX modules are built specifically as 32- or 64-bit versions. The version you load will depend on the host executable (the AutoCAD platform) you're working in, not on the OS. A 32-bit module will simply not load in a 64-bit version AutoCAD and vice-versa. The way most professional developers make sure the right version of their module is loaded, is to set up demand-loading keys appropriately from their installers, which AutoCAD uses to locate the appropriate modules and load them.

A module doesn't usually need to know whether it is 32- or 64-bit (and with polymorphic types in ObjectARX you should be able to build both versions off the same source code). That said - you might want to enable certain memory-intensive operations from your 64-bit modules but not from your 32-bit versions (for example), so one way is simply to declare a pointer and check its size (thanks to Gopinath Taget, from our DevTech team in San Rafael, for proposing this solution):

Adesk::IntPtr ptr;

int ptrSize = sizeof( ptr );

If ptrSize is 4, then you're in a 32-bit module - if ptrSize is 8, you're in a 64-bit module.

This could clearly also be exposed as a LISP-callable function (using acedDefun()), which is a solution for people who create their own ObjectARX modules but clearly not viable for people who don't.

So now back to our common scenario of people using LISP to load the correct version of an ObjectARX module: in the absence of a handy system variable, what do we do?

I thought about this for a while, and booted around some strange ideas such as using COM from LISP to query file attributes from AutoCAD binaries (yeech), and eventually decided that the best approach was simply to try to load a module, and if it fails, try a 64-bit specific name.

Here's the technique - you would use this function as a replacement for (arxload "myapp"):

(defun myarxload (fn / fn64)

  ;(princ (strcat "\nLoading " fn))

  (if

    (vl-catch-all-error-p

      (vl-catch-all-apply 'arxload (list fn))

    )

    (progn

      (setq fn64 (strcat fn "x64"))

      ;(princ (strcat "\nLoading " fn64))

      (if (findfile fn64)

        (arxload fn64)

      )

    )

  )

  (princ)

)

This code assumes a few things...

• We pass in the module name without the extension (as we append "x64" to the filename)
• We have used "x64" as a suffix for 64-bit versions of our modules (e.g. "AdskMyAppx64.arx")
  • I'm not aware of any convention for this... we simply use the same module names inside AutoCAD (which removes the need for code such as this, in any case)

I'd be very interested to hear the experiences and suggestions of readers of this blog on the subject. This topic has come up a few times and perhaps more of you have comments that would help.

Protecting intellectual property in AutoCAD application modules

This is an interesting topic – and one that I’m far from being expert in – so it would be great if readers could submit comments with additional information.

Intellectual property protection is a major concern for software developers, and issues that are seen today with .NET languages have been troubling AutoCAD developers since the introduction of AutoLISP.

So, what are these issues?

As a professional software developer, if you ship source-code to your customers there is substantial risk of it being borrowed or stolen for use in other unlicensed situations. This is true if you ship the actual source code used to build your modules, or if the “compiled” modules are actually not fully compiled, but (for example) stored in a CPU-independent, intermediate language that can quite easily be decompiled and have source code reconstituted or reverse-engineered (albeit without comments and usually without the original symbol names).

ObjectARX

This is much less of an issue with languages that are compiled to CPU-specific executable or machine code, such as for ObjectARX (and before it, ADS): the output from a C++ compiler does not, for instance, included any source code, unless you choose to ship debugging-related files such as PDBs (files which include information about function prototypes – information that is considered by some software providers as intellectual property in and of itself).

LISP

Intellectual property protection was historically a big issue with AutoLISP: in its original incarnation (i.e. before the integration of Visual LISP) LISP code was always interpreted by AutoCAD. Interpreted languages do not have a compilation phase – the code is “made sense of” by the interpreter at runtime. There were, however, things that could be done to obfuscate and protect the code before distribution. Most developers used these two tools on their code before shipping it:

• The Kelvinator – this nifty tool did a few things to obfuscate AutoLISP code:

Removed unnecessary whitespace

Stripped out comments

Mangled “internal” symbols (those not exposed as external variables or function names)

The Kelvinator was apparently written by Kelvin R. Throop, along with a number of other AutoLISP utilities. I wasn't around at the time (it was written in the late 80s and I joined in the mid 90s), so without ever having met Kelvin I do wonder whether he really exists, or whether this is just another use of the pseudonym used in a number of Science Fiction stories over the 20 or so years preceding the Kelvinator's development. This theory seems to be confirmed by this memo in the Autodesk File on John Walker's Fourmilab site. But if the real Kelvin R. Throop could step forward (or if someone else could confirm his existence or identity), then I'd willingly present my humble apologies. :-)

In terms of its functionality the Kelvinator was really quite effective: the only step that could be effectively reversed was the whitespace removal – by any LISP pretty printer. Getting meaningful comments & function/variable names back was nigh on impossible (although you could work out what certain variables were used for in a particular algorithm pretty easily).

• Protect.exe

This tool used very lightweight encryption to stop LISP files from being read by plain ASCII tools. From memory, I think a bit was reversed in each ASCII character – it was that level of encryption – which also meant that unprotection tools were also available (including one of the “c” versions of R13, if I remember correctly, which would print protected LISP as plain text to the AutoCAD text window :-).

Most professional developers at the time would Kelvinate and then Protect their LISP code before distributing it. Other options were to use LISP compilation – an early compiler was available pre-R13 versions, and Vital LISP (which later became Visual LISP) provided compilation capabilities.

Visual LISP changed the game for LISP developers: it allowed them to properly compile LISP into a protected format. Visual LISP supports two modes of compilation: each LISP file can be compiled to a format called FAS, which can be loaded directly into AutoCAD, and these can in turn be packaged into VLX modules. Originally Visual LISP allowed compilation to ARX, but these were essentially copies of the VL runtime with the compiled code stored as a resource, and clearly it’s more efficient to not force distribution of the runtime when it's in any case a standard component.

Over the years I've not come across anyone raising concerns about the security of the FAS or VLX formats, which leads me to believe they're a "secure" way to protect and distribute LISP modules (which ultimately means the effort required to make sense of the underlying logic of the application would be too high for it to be considered a useful practice for those developers unscrupulous enough to otherwise attempt it).

VB6

The VB6 IDE allows compilation to executable code – albeit code that requires a runtime component – and, once again, to the best of my knowledge this code is considered "secure” (see above note).

VBA

The VBA component embedded in AutoCAD allows password protection of DVB files (see the Protection tab on the Project Properties dialog in the VBA IDE). This password protection uses standard encryption to lock away source code from prying eyes. I understand that cracking tools are available for protected VBA modules, but there is no secret back-door – my team occasionally gets asked to help access source code stored in protected DVB files (as the person working on the code left the company without telling anyone the password for the module – or at least that’s what we’re told :-). Ultimately there’s nothing we can do to help in these cases, unfortunately. The cracking tools that are available seem to work on a brute force principle – they will cycle through possible passwords until one works – so if you want to really protect your code I’d recommend using a nice, long password.

.NET

So – life is basically good for LISP, ObjectARX, VBA and VB6 applications… what about .NET?

One of the major concerns shared by many .NET developers is around code security. Managed code gets compiled into assemblies that contain Microsoft Intermediate Language (MSIL) - basically CPU-independent instructions - in addition to metadata describing types, members and references to code in other assemblies. At runtime the MSIL gets converted to CPU-specific instructions by a just-in-time (JIT) compiler.

These assemblies are quite easy to disassemble into something resembling the original source code (albeit – once again – without comments or meaningful variable names, but internal function names and types get maintained).

Just as the Kelvinator was available for AutoLISP, there are obfuscation tools available for .NET languages. Visual Studio ships with one called the Dotfuscator Community Edition: this tools works on .NET assemblies, reducing the ease with which someone could reverse-engineer the source code.

At a basic level the tool strips namespace and member names - replacing them with symbols such as "a", "b", "c", ... - but it can only do so much to hide the logic of the code's execution, especially if there is no great dependency on functions and sub-routines (the more linear the code, the easier it is to understand, in my brief experience of analysing obfuscated code). That said, the Dotfuscator tool does have a number of options I haven't explored in depth, so it may well provide additional, helpful capabilities.

From taking a cursory look into what information is available by default in .NET assemblies, it's become clear to me that this should be an area of concern for anyone serious about protecting their source code investment. I strongly recommend that any professional software developer working with .NET spend time investigating obfuscation technologies for .NET assemblies. This guide, although a little dated, does seem to provide a fairly good background to .NET obfuscation, as well as presenting a number of different vendors' offerings.

In my next post I'm going to take a look at the Reflector tool - a very useful tool that can be used to disassemble .NET assemblies - and discuss how it might be used to help further one's understanding of .NET development.