Through the Interface

Metaprogramming with AutoCAD - Part 1

A recent comment on one of my F# articles got me thinking about this topic (thanks, Thomas! :-), so I thought I’d write a few posts on it. Next week is AU, and the week after that I’m attending a training class in Boston, so posts may be a little sparse over the coming weeks.

Metaprogramming  – according to the definition on Wikipedia – is the act of writing code that writes or manipulates other programs (or itself). But what is it really all about? The vast majority of programmers are actually metaprogramming without realizing it has such a fancy name.

To help understand metaprogramming, we’re going to focus on two ways of categorizing the various types of metaprogramming activity. Metaprogramming is usually either static or dynamic and homogeneous or heterogeneous (there are other classifications, but we’re not going to worry about those in this article).

• Static = at compile-time
• Dynamic = at runtime
• Homogeneous = the same output language is used as on the input
• Heterogeneous = a different output language is used to the input language

The most obvious form of metaprogramming is to create machine-code using a compiler (or even an interpreter) for a high-level language. This is a static, heterogeneous act of metaprogramming (although using an interpreter would presumably make this dynamic). Here are a few more interesting examples of metaprogramming I’ve used myself:

• C++ templates or pre-processor macros to generate lower-level code at compile-time
  • Static (compile-time) and heterogeneous (the generation language is different from the output language)
• Generation of a series of LISP expressions that are evaluated at runtime
  • Dynamic (runtime) and homogeneous (the generation and output languages are the same)
• Programmatic creation (perhaps using LISP, C#, VB(A) or C++) of an AutoCAD script that is then executed
  • Dynamic and heterogeneous
• Composing a SQL statement on-the-fly and using it to query a database
  • Dynamic and heterogeneous

The focus of this series of posts is on dynamic metaprogramming, which allows the modification of code at runtime (rather than compile-time). A further, somewhat more complex, example of dynamic metaprogramming is to redefine functions at runtime (something that’s possible with LISP, for instance), which allows applications to evolve, such as when developing expert systems that “learn” over time.

LISP was really one of the early programming environments that enabled metaprogramming, primarily through its ability to evaluate expression using (eval) and to redefine functions at runtime using (defun). This has been immensely valuable to AutoLISP programmers over the years. When AutoLISP was first introduced it was purely an interpreted language, so dynamic metaprogramming was provided pretty much automatically. In order for metaprogramming to work after the introduction of Visual LISP (which is fundamentally a compiled environment, albeit to an intermediate language), a runtime component supporting dynamic compilation was needed and provided. Very little change was needed in AutoLISP code, although in some rare cases (defun-q) now needs to be used, if it’s important to provide access to the internal representation of functions.

We’ll see this is a common thread for dynamic metaprogramming: by definition you either need to be working in an interpreted environment or will need to have a runtime component available that supports some kind of compilation (probably JIT). Visual LISP provides this, as does VBA and .NET (via the CLR).

Back to AutoLISP: one very common activity is to interpret a string using (read) and then call (eval) on it. The string may have been stored in a drawing, a text file, an external database, or generated on-the-fly. For example:

Command: (eval (read "(* 5 (getvar \"ZOOMFACTOR\"))"))

300

VBA also has native support for dynamic metaprogramming via the Eval() function:

Eval "MsgBox ThisDrawing.Name"

VB6 doesn't have direct support for Eval(), but it seems you can make use of it either by embedding a Script Control or by calling across to the VBA runtime (Googling "VB6 Eval" returned a number of options). I don't know whether it's possible to evaluate and make use of AutoCAD-specific variables - such as ThisDrawing - when using these techniques, however.

Metaprogramming with .NET is not quite so automatic, but is altogether possible, as I’ll show in my next post.

Protecting intellectual property in AutoCAD application modules

This is an interesting topic – and one that I’m far from being expert in – so it would be great if readers could submit comments with additional information.

Intellectual property protection is a major concern for software developers, and issues that are seen today with .NET languages have been troubling AutoCAD developers since the introduction of AutoLISP.

So, what are these issues?

As a professional software developer, if you ship source-code to your customers there is substantial risk of it being borrowed or stolen for use in other unlicensed situations. This is true if you ship the actual source code used to build your modules, or if the “compiled” modules are actually not fully compiled, but (for example) stored in a CPU-independent, intermediate language that can quite easily be decompiled and have source code reconstituted or reverse-engineered (albeit without comments and usually without the original symbol names).

ObjectARX

This is much less of an issue with languages that are compiled to CPU-specific executable or machine code, such as for ObjectARX (and before it, ADS): the output from a C++ compiler does not, for instance, included any source code, unless you choose to ship debugging-related files such as PDBs (files which include information about function prototypes – information that is considered by some software providers as intellectual property in and of itself).

LISP

Intellectual property protection was historically a big issue with AutoLISP: in its original incarnation (i.e. before the integration of Visual LISP) LISP code was always interpreted by AutoCAD. Interpreted languages do not have a compilation phase – the code is “made sense of” by the interpreter at runtime. There were, however, things that could be done to obfuscate and protect the code before distribution. Most developers used these two tools on their code before shipping it:

• The Kelvinator – this nifty tool did a few things to obfuscate AutoLISP code:

Removed unnecessary whitespace

Stripped out comments

Mangled “internal” symbols (those not exposed as external variables or function names)

The Kelvinator was apparently written by Kelvin R. Throop, along with a number of other AutoLISP utilities. I wasn't around at the time (it was written in the late 80s and I joined in the mid 90s), so without ever having met Kelvin I do wonder whether he really exists, or whether this is just another use of the pseudonym used in a number of Science Fiction stories over the 20 or so years preceding the Kelvinator's development. This theory seems to be confirmed by this memo in the Autodesk File on John Walker's Fourmilab site. But if the real Kelvin R. Throop could step forward (or if someone else could confirm his existence or identity), then I'd willingly present my humble apologies. :-)

In terms of its functionality the Kelvinator was really quite effective: the only step that could be effectively reversed was the whitespace removal – by any LISP pretty printer. Getting meaningful comments & function/variable names back was nigh on impossible (although you could work out what certain variables were used for in a particular algorithm pretty easily).

• Protect.exe

This tool used very lightweight encryption to stop LISP files from being read by plain ASCII tools. From memory, I think a bit was reversed in each ASCII character – it was that level of encryption – which also meant that unprotection tools were also available (including one of the “c” versions of R13, if I remember correctly, which would print protected LISP as plain text to the AutoCAD text window :-).

Most professional developers at the time would Kelvinate and then Protect their LISP code before distributing it. Other options were to use LISP compilation – an early compiler was available pre-R13 versions, and Vital LISP (which later became Visual LISP) provided compilation capabilities.

Visual LISP changed the game for LISP developers: it allowed them to properly compile LISP into a protected format. Visual LISP supports two modes of compilation: each LISP file can be compiled to a format called FAS, which can be loaded directly into AutoCAD, and these can in turn be packaged into VLX modules. Originally Visual LISP allowed compilation to ARX, but these were essentially copies of the VL runtime with the compiled code stored as a resource, and clearly it’s more efficient to not force distribution of the runtime when it's in any case a standard component.

Over the years I've not come across anyone raising concerns about the security of the FAS or VLX formats, which leads me to believe they're a "secure" way to protect and distribute LISP modules (which ultimately means the effort required to make sense of the underlying logic of the application would be too high for it to be considered a useful practice for those developers unscrupulous enough to otherwise attempt it).

VB6

The VB6 IDE allows compilation to executable code – albeit code that requires a runtime component – and, once again, to the best of my knowledge this code is considered "secure” (see above note).

VBA

The VBA component embedded in AutoCAD allows password protection of DVB files (see the Protection tab on the Project Properties dialog in the VBA IDE). This password protection uses standard encryption to lock away source code from prying eyes. I understand that cracking tools are available for protected VBA modules, but there is no secret back-door – my team occasionally gets asked to help access source code stored in protected DVB files (as the person working on the code left the company without telling anyone the password for the module – or at least that’s what we’re told :-). Ultimately there’s nothing we can do to help in these cases, unfortunately. The cracking tools that are available seem to work on a brute force principle – they will cycle through possible passwords until one works – so if you want to really protect your code I’d recommend using a nice, long password.

.NET

So – life is basically good for LISP, ObjectARX, VBA and VB6 applications… what about .NET?

One of the major concerns shared by many .NET developers is around code security. Managed code gets compiled into assemblies that contain Microsoft Intermediate Language (MSIL) - basically CPU-independent instructions - in addition to metadata describing types, members and references to code in other assemblies. At runtime the MSIL gets converted to CPU-specific instructions by a just-in-time (JIT) compiler.

These assemblies are quite easy to disassemble into something resembling the original source code (albeit – once again – without comments or meaningful variable names, but internal function names and types get maintained).

Just as the Kelvinator was available for AutoLISP, there are obfuscation tools available for .NET languages. Visual Studio ships with one called the Dotfuscator Community Edition: this tools works on .NET assemblies, reducing the ease with which someone could reverse-engineer the source code.

At a basic level the tool strips namespace and member names - replacing them with symbols such as "a", "b", "c", ... - but it can only do so much to hide the logic of the code's execution, especially if there is no great dependency on functions and sub-routines (the more linear the code, the easier it is to understand, in my brief experience of analysing obfuscated code). That said, the Dotfuscator tool does have a number of options I haven't explored in depth, so it may well provide additional, helpful capabilities.

From taking a cursory look into what information is available by default in .NET assemblies, it's become clear to me that this should be an area of concern for anyone serious about protecting their source code investment. I strongly recommend that any professional software developer working with .NET spend time investigating obfuscation technologies for .NET assemblies. This guide, although a little dated, does seem to provide a fairly good background to .NET obfuscation, as well as presenting a number of different vendors' offerings.

In my next post I'm going to take a look at the Reflector tool - a very useful tool that can be used to disassemble .NET assemblies - and discuss how it might be used to help further one's understanding of .NET development.

COM vs. .NET in AutoCAD

This was a topic suggested by Scott Underwood (thanks, Scott! :-) to look at COM vs. NET and go through their respective advantages and disadvantages…

This is really an interesting discussion topic, and one that I’d like people to help turn into an interesting discussion. I can certainly talk about the differences and pros/cons of the two technologies from my own memory/experience/perspective, but others will have things to say on this, I’m sure... please feel free to do so! :-)

Rather than going into low-level detail on either COM or .NET, I’d suggest looking into their respective Wikipedia sites. Both do a fair job of listing the criticisms made about the respective technologies, but not all will apply here (the .NET Framework is part of the AutoCAD install, so availability/deployment is not such a big issue for us, for instance).

So let's start with some generally AutoCAD API background…

Both COM and .NET APIs (and actually also AutoLISP, since it's re-implementation as Visual LISP back in R14.01) are implemented via ObjectARX – they are basically ARX modules loaded into AutoCAD that expose COM and managed interfaces respectively. That said, as more core AutoCAD development happens using .NET, it’s no longer necessarily true that managed interfaces will always be exposed through (and available in) ObjectARX. The decision about which technology to use to expose an API is driven by its market requirements.

Now a little more specific history on each of the technologies…

COM Automation (originally OLE Automation, changed to ActiveX Automation, and now it’s either COM Automation or just Automation)

AutoCAD’s COM Automation API was first introduced with VBA back in R14.01. At the time it was pretty revolutionary, as it was an API that was implemented via a technology that genuinely allowed developers to choose their own development technology (as long as it was Visual Basic… just kidding ;-). It was also the technology that enabled some very clever features in AutoCAD: the Properties Palette uses COM Automation to query and edit properties of objects, for instance. QSELECT is another one.

One of the big problems with COM is the effort needed to expose COM interfaces: the COM Automation standard supports a relatively small set of data-types (unless you make nearly everything its own object, and that just gets unmanageable), which means that complex C++ interfaces need to be fundamentally re-designed (often flattened) to be exposed through COM.

.NET

When AutoCAD was first built using Visual Studio .NET (during the AutoCAD 2004 timeframe), the door was opened to expose a managed API. While this was prototyped on AutoCAD 2004, it first made it into the shipping product as part of AutoCAD 2005. Exposing a managed interface is much easier than a COM Automation interface... in fact the AutoCAD team has managed to semi-automate the process of exposing its managed interfaces. This definitely helps us keep the managed API in synch with ObjectARX, where - as mentioned earlier - much of it is exposed from.

Incidentally, a managed API to AutoCAD also enables Autodesk to do more internal development using .NET, rather than all of it being in unmanaged C++ (much of its feature development work is still done in C++/ObjectARX, but that's changing). It enables us to take advantage of many of the advantages of the .NET Framework, such as the tools around creating user interfaces, accessing data, etc.

And now let’s compare some of the specific differences, stepping through some typical API comparison criteria, one-by-one... (I would have liked to make a little matrix showing this, but felt the need to flesh out the explanations.)

Performance

Both technologies are comparable in terms of speed of execution – neither introduce much execution overhead when compared with ObjectARX (unless you’re using COM out-of-process – more on that later).

Future-proofness (if that’s even a word :-)

COM is being extended as per the needs of various AutoCAD features (Properties Palette support, etc.), but you will see more rapid expansion of .NET capabilities. Additionally Microsoft is 100% behind .NET, and I genuinely believe it has a much brighter future.

Simplicity

This is subjective, but overall I would say the COM object model is simpler to learn. That doesn’t mean I prefer it, though. (How’s that for subjective? :-)

Power

.NET is more extensive in terms of its level of API exposure and also has more powerful platform capabilities (although that's more when you get into comparing VB6 with VB.NET).

Interoperability

COM environments such as VBA/VB6 can use all sorts of COM components and even DLL/EXE exports, but from .NET you can use much, much more (COM, .NET, native C++ etc.)

Support in shipping AutoCAD versions

COM is available throughout our shipping versions of AutoCAD while .NET is not quite there yet. Although it won't be long before all supported versions of AutoCAD have a managed API, our .NET implementation evolved substantially between 2005 and 2006, and AutoCAD 2005 will still be supported for some time. If you need to support multiple versions, this is something you need to be aware of.

IPC (Inter-Process Communication)

COM’s big advantage – and frankly the main reason I use it at all – is that it was designed to be used across processes. While .NET Remoting is possible with some applications, AutoCAD’s managed interface was not designed to work across the process boundary (just as ObjectARX was not).

Conclusion

If I had to learn a new API for AutoCAD at this stage (and didn't know any of them), I would choose .NET. It may be more complex to learn, but it is more extensive, it has better long-term potential both as an API and a programming environment (in terms of support from Microsoft and Autodesk), and frankly you can make very easily make use of AutoCAD's COM Automation interface from a .NET environment. It has very good coverage of the overall ObjectARX feature-set, and much of what isn't exposed can be accessed using P/Invoke. If you need to drive AutoCAD from an external executable, then I’d suggest using COM to drive AutoCAD out-of-process, loading a .NET component in-process to do the heavy lifting.

The main caveat about all this is around platform support. Until your customer(s) all use a version of AutoCAD that exposes the API set you need, you might well need to invest time developing with other technologies. This was also an issue when ADS, ObjectARX and COM/VB(A) were introduced (and probably even AutoLISP, although I don't go back quite that far).

Techniques for calling AutoCAD commands programmatically

It's quite common to want to call commands from one or other of AutoCAD's programming environments. While it's cleanest (from a purist's perspective) to use an API to perform the task you want, the quickest way - and the one which will appeal to the pragmatists (and the lazy) among us is often to call a sequence of commands. And there are, of course, a few places in AutoCAD where APIs have not been exposed, so scripting commands is the only way to achieve what you want.

Let's take the simple example of adding a line. Here's what you'd do from the different environments from a low-level API perspective:

• LISP - create an association list representing the entity and then (entmake) it
• VBA (or COM) - get the ModelSpace object from the active drawing and call AddLine (the COM API is probably the simplest in that respect)
• .NET and ObjectARX - open the block table and then the model-space block table record, create a new line object and append it to the model-space (and to the transaction, if you're using them), closing the various objects along the way

Having first started coding for AutoCAD with LISP (for R10), I know that the simplest way to do what you want from that environment is to call the LINE command, passing in the start and end points:

(command "_LINE" "0,0,0" "100,100,0" "")

LISP is great in that respect: as you're not able to define native commands (only LISP) functions, it's perfectly acceptable to use it to script commands to do what you want, rather than rely on low-level APIs.

ObjectARX in particular has potential issues with respect to defining native commands calling commands, as AutoCAD is only "re-entrant" up to 4 levels. Without going into specifics, it's basically best to avoid calling commands using acedCommand() from ObjectARX, unless the command is registered as a LISP function using acedDefun().

While you do have to be careful when calling commands from VBA or ObjectARX, there are a few options available to you.

ObjectARX

• ads_queueexpr()
  • This old favourite is intended to be used from acrxEntryPoint() to execute a sequence of commands after (s::startup) has been called from LISP (as you are not able to use acedCommand() from this context)
  • You need to declare it yourself (extern "C" void ads_queueexpr( ACHAR *);) before use
  • It has been unsupported as long as I can remember, but is widely-used and mentioned in the Tips & Techniques section of the ObjectARX Developer's Guide
• AcApDocManager::sendStringToExecute()
  • This function has the advantage of a few more options being available as arguments, mainly around where to execute the string (which document, and whether it be activated), and whether to echo the string to the command-line
• ::SendMessage()
  • This is a standard Win32 platform API and so can, in effect, be used to drive AutoCAD from an external client. It uses a structure to pass the string that is often a bit tricky to set up (it became a migration issue when we switched to Unicode, for example)
• IAcadDocument::SendCommand()
  • This COM method is the only way (other than acedCommand() or acedCmd()) to execute a command synchronously from AutoCAD (and even then it may not be completely synchronous if requiring user input)
• acedCommand()
  • This is the ObjectARX equivalent to (command), and is genuinely synchronous. Unfortunately (as mentioned earlier) there are issues with using it directly from a natively-registered command, so I'd recommend only using it from acedDefun()-registered commands (see the ObjectARX documentation and the below sample for more details)

VBA (some of which also applies to VB)

• ThisDrawing.SendCommand
  • This is the same as IAcadDocument::SendCommand() from C++
• SendKeys
  • This is just a simple technique to send key-strokes to the command-line
• SendMessage
  • This is just the Win32 API mentioned above, but declared and called from VB(A)

So, now for some sample code...

ObjectARX sample code

The first can be dropped into an ObjectARX Wizard-defined project (I used Visual Studio 2005 and ObjectARX 2007). You'll need to make sure "COM-client" is selected and you name your project "SendingCommands" (or you search and replace to change the name in the below code to something you prefer).

The code creates points along a line (from 0,0 to 5,5), using different techniques to send the command to AutoCAD. I would, of course, use the proper ObjectARX APIs to do this (creating an AcDbPoint etc.) - I just used this as an example of a command that could be sent.

It creates the first point on load, using ads_queueexpr(), and defines commands (TEST1, TEST2, TEST3 and TEST4) for the subsequent tests (the last being an acedDefun()-registered command).

#include "StdAfx.h"

#include "resource.h"

#define szRDS _RXST("Adsk")

extern "C" void ads_queueexpr( ACHAR *);

//----- ObjectARX EntryPoint

class CSendingCommandsApp : public AcRxArxApp {

public:

  CSendingCommandsApp () : AcRxArxApp () {}

  virtual AcRx::AppRetCode On_kInitAppMsg (void *pkt) {

    // You *must* call On_kInitAppMsg here

    AcRx::AppRetCode retCode =AcRxArxApp::On_kInitAppMsg (pkt) ;

    return (retCode) ;

  }

  virtual AcRx::AppRetCode On_kUnloadAppMsg (void *pkt) {

    // You *must* call On_kUnloadAppMsg here

    AcRx::AppRetCode retCode =AcRxArxApp::On_kUnloadAppMsg (pkt) ;

    return (retCode) ;

  }

  virtual AcRx::AppRetCode On_kLoadDwgMsg(void * pkt) {

    AcRx::AppRetCode retCode =AcRxArxApp::On_kLoadDwgMsg (pkt) ;

    ads_queueexpr( _T("(command\"_POINT\" \"1,1,0\")") );

    return (retCode) ;

  }

  virtual void RegisterServerComponents () {

  }

public:

  // - AdskSendingCommands._SendStringToExecTest command (do not rename)

  static void AdskSendingCommands_SendStringToExecTest(void)

  {

    acDocManager->sendStringToExecute(curDoc(), _T("_POINT 2,2,0 "));   

  }

  static void SendCmdToAcad(ACHAR *cmd)

  {

    COPYDATASTRUCT cmdMsg;

    cmdMsg.dwData = (DWORD)1;

    cmdMsg.cbData = (DWORD)(_tcslen(cmd) + 1) * sizeof(ACHAR);

    cmdMsg.lpData = cmd;

    SendMessage(adsw_acadMainWnd(), WM_COPYDATA, NULL, (LPARAM)&cmdMsg);

  }

  // - AdskSendingCommands._SendMessageTest command (do not rename)

  static void AdskSendingCommands_SendMessageTest(void)

  {

    SendCmdToAcad(_T("_POINT 3,3,0 "));

  }

  // - AdskSendingCommands._SendCommandTest command (do not rename)

  static void AdskSendingCommands_SendCommandTest(void)

  {

    try {

      IAcadApplicationPtr pApp = acedGetIDispatch(TRUE);

      IAcadDocumentPtr pDoc;

      pApp->get_ActiveDocument(&pDoc);

      pDoc->SendCommand( _T("_POINT 4,4,0 ") );

    }

    catch(_com_error& e) {

      acutPrintf(_T("\nCOM error: %s"), (ACHAR*)e.Description());

    }

  }

  // ----- ads_test4 symbol (do not rename)

  static int ads_test4(void)

  {

    acedCommand(RTSTR, _T("_POINT"), RTSTR,_T("5,5,0"), RTNONE);

    acedRetVoid();

    return (RSRSLT);

  }

} ;

// ----------------------------------------------------------

ACED_ARXCOMMAND_ENTRY_AUTO(CSendingCommandsApp, AdskSendingCommands, _SendStringToExecTest, TEST1, ACRX_CMD_TRANSPARENT, NULL)

ACED_ARXCOMMAND_ENTRY_AUTO(CSendingCommandsApp, AdskSendingCommands, _SendMessageTest, TEST2, ACRX_CMD_TRANSPARENT, NULL)

ACED_ARXCOMMAND_ENTRY_AUTO(CSendingCommandsApp, AdskSendingCommands, _SendCommandTest, TEST3, ACRX_CMD_TRANSPARENT, NULL)

ACED_ADSCOMMAND_ENTRY_AUTO(CSendingCommandsApp, test4, true)

// ----------------------------------------------------------

IMPLEMENT_ARX_ENTRYPOINT(CSendingCommandsApp)

VBA sample code

This sample defines VBA macros that create points from 6,6 to 8,8, using techniques that mirror the ones shown in the ObjectARX sample.

Option Explicit On

Private Const WM_COPYDATA = &H4A

Private Type COPYDATASTRUCT

    dwData As Long

    cbData As Long

    lpData As String

End Type

Private Declare Function SendMessage Lib "user32" Alias _

  "SendMessageA" (ByVal hwnd As Long, ByVal wMsg As Long, ByVal _

  wParam As Long, ByVal lParam As Any) As Long

Public Sub SendMessageToAutoCAD(ByVal message As String)

    Dim data As COPYDATASTRUCT

    Dim str As String

    str = StrConv(message, vbUnicode) 'converts to Unicode

    data.dwData = 1

    data.lpData = str

    data.cbData = (Len(str) + 2)

    SendMessage(ThisDrawing.Application.hwnd, WM_COPYDATA, 0, data)

End Sub

Public Sub Test4()

    ThisDrawing.SendCommand("_point 6,6,0 ")

End Sub

Public Sub Test5()

    SendKeys("_point 7,7,0 ")

End Sub

Public Sub Test6()

    SendMessageToAutoCAD("_point 8,8,0 ")

End Sub