Through the Interface

Debugging into AutoCAD’s .NET API layer using Reflector – Part 2

In the first post in this series we looked at the basic (and free) capabilities of the upcoming version 6 of .NET Reflector. Today we’ll look at the “Pro” capabilities, which integrate decompilation with debugging of 3rd party assemblies.

To get started, we should see a new “.NET Reflector” toolbar inside our Visual Studio installation. On it, we should see a “Choose Assemblies to Debug…” option:

This option launches an analysis of our currently open solution, to understand the assemblies that have been referenced inside it:

One very important thing to note: for Reflector to work, you must have assembly references to the versions of acmgd.dll and acdbmgd.dll found in AutoCAD’s Program Files folder. If you’re using the “liposuctioned” assemblies provided in the ObjectARX SDK (which are absolutely the right ones to use under normal circumstances) then you’ll get an error such as “Object reference not set to an instance of an object”, at least with the Beta version of this tool I tried.

To get started with the decompilation, we select AcMgd and AcDbMgd from the assembly list:

Selecting OK causes the decompilation process to start on these assemblies:

Once complete, we see a notice describing the results. For the AutoCAD 2010 assemblies there were 28 methods that failed to decompile (and 19,390 which decompiled successfully):

Looking at the details, we can see that the majority of the non-decompiled (for want of a better word :-) methods belong to the ObjectId class (whether due to an optimization we make on that class or a bug in Reflector’s decompilation algorithm):

The decompilation has added about 33 Mb of data to Reflector’s “Debug Store” (which appears to be under C:\Users\username\AppData\Local\Red Gate\.NET Reflector 6\Cache, at least on my system).

Now we can go ahead and debug as normal and see the results. Let’s start by adding a breakpoint to our Extend method. Once it hits, we can step into the code:

Firstly we’ll see some code that shows us retrieving the StartPoint property of a Line (which I haven’t shown here), and then we get to debug the use of the subtraction operator between a point and a vector (which we can see subtracts the coordinate values of the vector from those of the point before returning the point):

And next we get to see that the Extend call is simple a call through to an unmanaged ObjectARX interface (which happens to be AcDbCurve::extend(), not that this is in any way obvious from the decompiled code):

Fun stuff! I can see this being of great interest to people who like to get under the hood of software components, although clearly a lot will depend on the cost at which the tool is made available (and about this I have no clue).

Debugging into AutoCAD’s .NET API layer using Reflector – Part 1

As many of you will know, by now, I’m a big fan of Reflector. I’ve used it for quite some time, since its early days as a piece of freeware developed by Lutz Roeder, and overall it’s proven to be an extremely valuable tool. When a UK-based company, Red Gate Software, acquired the Reflector technology back in 2008, I was curious as to their strategy for monetizing it (and it seems I wasn’t alone in speculating on the direction the tool would take as a commercial offering). Well, that’s now starting to become clear with the Beta release of .NET Reflector Pro.

Here’s the tag-line on Red Gate’s website for this new version:

.NET Reflector Pro integrates .NET Reflector into Visual Studio to allow you to seamlessly debug into third-party code and assemblies, even if you don't have the source code for them. You can now treat decompiled assemblies much like your own code: you can step through the assemblies and use all the debugging techniques that you would use on your own code.

This blog article on Red Gate’s Simple-Talk site contains some more interesting details on this version’s capabilities. Here are the two main features:

• Letting you jump to Reflector from your code (this is free!)
• Debugging decompiled code using the visual studio debugger (this is Reflector Pro - so will cost money)

Both of these items look pretty interesting, and – knowing that the .NET API for AutoCAD is unobfuscated, as it’s mostly a fairly thin layer on top of ObjectARX – the second (paid) feature has the potential to be of great interest to AutoCAD .NET developers.

So I decided to put the tool through its paces. After downloading the most recent external Beta (build 892), and extracting it to my hard-drive, I launched Reflector.

On the Tools menu, we see the Integration Options menu item:

In the options, themselves, we can see that the tool has detected the installed versions of Visual Studio on my system and has integrated itself appropriately with them:

Now, as long as we’ve previously loaded AutoCAD’s managed assemblies inside Reflector (File –> Open, and select “acmgd.dll” and “acdbmgd.dll” from our AutoCAD installation folder), we should now be able to jump directly to the implementation of a particular method from the Visual Studio window. Let’s take the example of the method looked used in the last post, the Extend method we called on a Line:

And when we select it, the integration takes us to the Curve class in Reflector:

From here it’s a manual operation to expand the class and select “Extend(Boolean, Point3d) : Void” from the list of methods (although I’m curious why it didn’t manage to get there itself – perhaps a minor Beta quirk).

And then we can, of course, use the standard Reflector capabilities to look into the code:

Which admittedly only shows us (in this case, at least) an obscure piece of code which is actually calling through to the underlying, unmanaged, ObjectARX API:

In the second part of this two-part series, we’ll take a look at the “Pro” features of this new .NET Reflector version: the actual debugging integration.

DevLabs reminder

Just a quick reminder about the DevLabs my team is running, the first of which is being held next week in our UK office.

Here’s some additional information that was sent out in a recent technical newsletter for ADN members:

If you don’t have time to attend a training class, but you value the opportunity to work on your code with the help of an Autodesk expert, then DevLab is for you. Bring your laptop to our FREE DevLab and work on your code as you normally – the difference is that you have a team of DevTech experts (the same people who provide API support, training and consulting to ADN members) available to review your code, answer your questions, and make suggestions. Stay at DevLab for a week, or just for one or two days – you’re free to come and go as you please. (And did we mention it was FREE?).

If you are confused about what DevLabs are all about, here are some sample use cases:

1. You have new requirements for your product but you are unsure of what direction to take or which technology to use. You need to ask questions and then take time to think once you get the answers in order to decide what direction you want to take. Setting up a full Mentoring engagement is too much for your particular requirements. Perhaps you have a few questions that you need to describe in person early on and once answered, you can continue your investigation and your everyday work; but then again, after some time in carrying on with your work, you may get stuck and require further consultation. Feel free to sit in a DevLabs session and either work as normal or ask questions.
2. You are developing new features in your product, coding as you go. You know how to do most of the development required, but you just get stuck on certain details because you are unfamiliar with the technology. These things can take a lot of time to research! You know that if you ask a question and get an answer, you will be able to continue programming until you get stuck again and then require more direct consultation.
3. You are looking for a chance to meet and share ideas with Autodesk experts and other programmers, but you are very busy and just can’t justify spending time at a conference or specific training class. You want to be able to get on with your normal work, but at the same time, benefit from being able to discuss issues that you or other developers encounter. It’s totally up to you when you ask for help, or listen in. If you just sit and work on your own thing for the whole time, that’s fine.

Our trial DevLabs are:

• Farnborough, UK
• June 1–5, AutoCAD and Autodesk Inventor APIs
• San Francisco, USA
• June 22–26, AutoCAD APIs
• Bangalore, India
• September 8–11, Geospatial and Revit APIs
• September 14-17, AutoCAD and Autodesk Inventor APIs
• Beijing, China
• August 17–21, AutoCAD and Autodesk Inventor APIs
• August 24–28, Geospatial and Revit APIs

Email dev-consult@autodesk.com for more information or to register.

Introducing DevLabs

My team is trying out a new concept over the coming months, something we’re calling DevLabs. The idea is that we schedule a period where members of my team are available in a particular Autodesk office to sit down with people working on their development projects. (The concept is new for us, at least: I know Google does something similar with their Hackathons.)

Attendees can plan on dropping in for a just a day or for longer – even the whole week, if they so choose - to spend time working through development issues with members of my team. This is not a replacement for API training, of course, as people in the room will each have their own requirements, although we will try to send enough members of my team to each location to make sure the people attending get enough attention (assuming they’re not expecting us to focus on them exclusively).

You will need to bring a portable computer with your development environment on it (up to you whether you bring a netbook, notebook, desktop or smart-phone – the main thing is that you have access to your code and whatever compiler is needed for it :-) and as you code you can ask members of the DevTech team how to solve certain problems, get them to review your code or to make suggestions for alternative implementations. You may even get input from other DevLab attendees, too.

There’s no charge for attending, one of the reasons we’re holding the events in Autodesk offices where we have local staff. Attendees will only need to make their own arrangements for travel, accommodation and meals. And during this trial period we’re keeping the DevLabs open to all: there’s no requirement to be a member of the Autodesk Developer Network, for instance.

Here’s the date, location and product focus for each of the trial events:

• San Francisco, USA
  • June 22-26, AutoCAD APIs
• Farnborough, UK
  • June 1-5, AutoCAD and Autodesk Inventor APIs
• Bangalore, India
  • September 8–11, Geospatial and Revit APIs
  • September 14-17, AutoCAD and Autodesk Inventor APIs
• Beijing, China
  • August 17-21, AutoCAD and Autodesk Inventor APIs
  • August 24-28, Geospatial and Revit APIs

If you have any questions about these DevLabs or would like to register, please email us at dev-consult@autodesk.com.

Tired of not being able to NETUNLOAD from AutoCAD? "Edit and Continue" to the rescue!

A question came in on a previous post:

Hello, I write applications for Autocad Map and Civil3d platforms, mostly with ObjectARX. I would like to do more with .NET but so far the main reason preventing this is not having the NETUNLOAD command.. With arx I can just arxunload and arxload the application for modifications in a second. But with .NET I have to restart the heavy environment and do all kinds of lengthy initializations before being able to try even small changes in code, this can take a minute or more.. Maybe it is possible to create an utility, for development purposes, to unload a .net assembly from Autocad ?

NETUNLOAD has been requested many times by developers, but unfortunately would require a significant change for the .NET API - one it's unclear there'd be significant benefit in making. The root of the situation is that .NET assemblies cannot be unloaded from an AppDomain.

To implement a NETUNLOAD command, we would have to host each assembly in a separate AppDomain and then destroy the AppDomain to unload it. It's altogether possible to implement your own bootstrapper assembly that does just this: I'm going to give that a try, to see how it works, but in the meantime I thought I'd point out (or remind you of) a capability that to greatly reduces the need to continually unload modules from AutoCAD: Visual Studio's Edit and Continue.

Edit and Continue has been around since VB6, although you may not have looked at it in recent versions of Visual Studio. I personally didn't start finding it usable again until Visual Studio 2005 (and its Express Editions). If you'd previously turned your back on it, I suggest taking another look.

To enable Edit and Continue, start by editing the Debugging options (accessed via Tools -> Options in the Visual Studio editor):

If you attempt to edit the code before the module has been loaded, you'll get this error:

And if you try to edit the code before a breakpoint has hit, you'll get this altogether more obscure error:

This message wasn't very helpful - at least not to me. My code wasn't running, as such (although I agree that someone's code was :-) and the setting mentioned was set correctly. Anyway - breaking into your code allows you to then edit the code, which should be obvious by a message on the application status-bar that tells you when the "continue" piece is successful:

I'd be curious to hear about others opinions of Edit and Continue: does it meet your debugging needs, or do you still see NETUNLOAD as important functionality? (There are clearly still areas - unrelated to debugging - where it might be useful to unload .NET assemblies. To be clear, I'd be happy to receive input on those areas, also.)

Advanced Visual Studio debugging: how to stop stepping into certain functions

While debugging it's sometimes very frustrating to find yourself repeatedly stepping into an irrelevant function. For instance, complex statements pretty commonly include object constructors etc. that you know function perfectly well, but the debugger routinely takes you into them.

The Visual Studio IDE has an undocumented (and unsupported) mechanism to help with this. During the VC 6.0 timeframe it was implemented via our old friend the autoexp.dat file (see my previous post on this), in a special section called [ExecutionControl]. Since VC 7.0 this has been moved to the Registry.

For VC 7.0 and 7.1, it was in the HKCU (HKEY_CURRENT_USER) section:

HKCU\Software\Microsoft\VisualStudio\7.0\NativeDE\StepOver
HKCU\Software\Microsoft\VisualStudio\7.1\NativeDE\StepOver

For VC 8.0 (Visual Studio 2005), it shifted across to HKLM (HKEY_LOCAL_MACHINE):

HKLM\SOFTWARE\Microsoft\VisualStudio\8.0\NativeDE\StepOver

Inside this key you will need to create string values that specify the functions and class methods to exclude (or include) while stepping through code in the debugger. For VC7.0/7.1 you needed to use a numeric identifier for the name, but with VC8.0 you can thankfully use something more meaningful:

Name                Value
AcDbOP cons    AcDbObjectPointer\<.*\>\:\:AcDbObjectPointer.*=NoStepInto
App init              InitApplication=NoStepInto

[ Note: I would have used something even more meaningful than AcDbOP above, but I wanted the text to all fit on one line for clarity... :-) ]

The first part of the string value needs to be defined as a regular expression: the text I'm looking for in the first instance is really "AcDbObjectPointer<*>::AcDbObjectPointer*", but I had to use an escape character to prefix some of the symbols. Check MSDN for more information on regular expressions.

This is also the string as you would enter it via the Registry Editor - if you're using a .reg file then you'll need to double-up the slashes.

The second part of the string value is likely to be "=NoStepInto" or "=StepInto", depending on whether you are telling the debugger to exclude or include the locations defined by the expression. If you want to get really clever, you can use wildcards to exclude all the functions of a particular class, and then specifically re-include the ones that interest you. This CodeGuru forum post shows some good examples of that. I'd also recommend reading this entry from Andy Pennell's blog.

Incidentally, you don't have to restart Visual Studio for the changes to be picked up - both autoexp.dat and this section of the Registry are read in as a debug session is started (although the autoexp.dat timestamp needs to be more recent than the end of the last debugging session, interestingly enough... so if you make changes and save while a debug session is running, they won't be picked up in the next session unless a save is done between the sessions <phew!>).

When you come to execute your code, you should find that it no longer steps into the constructor of any class defined using the AcDbObjectPointer template (such as AcDbObjectPointer<AcDbViewportTableRecord>).

[ Side note: This template is one of the ObjectARX "smart pointers", and are a great way to handle the opening and closing of persistent ObjectARX objects. I won't go into detail here, but they are well worth checking out in the ObjectARX SDK documentation. ]

When I say "step into", I literally mean you would need to be stepping through the code using the debug toolbar or F11 - execution still stops in that particular function if you set a break-point there.

Advanced Visual Studio debugging: automatic expansion of watched variables

I've been using Visual C++ (and afterwards Visual Studio) since it was 16-bit, back in version 1.52. OK, maybe that's not so long ago, relatively (11 short years), but the point is that in spite of having followed the Visual Studio technology over this period, I've so far been completely unaware of the autoexp.dat file.

This feature of the Visual Studio was brought to my attention by Ahsan Ali, a programmer in the Inventor Engineering team who was based over in Bangalore at the same time I was (we had both previously worked in the US - he had come across from Tualatin while I had moved there from San Rafael). During a recent technology discussion, Ahsan shared some information with our Bangalore-based team on some advanced debugging techniques, and I thought it would be a great topic for this blog.

The autoexp.dat file's default location (for Visual Studio 2005) is:

C:\Program Files\Microsoft Visual Studio 8\Common7\Packages\Debugger\autoexp.dat

It's basically a text file that informs the Visual Studio IDE which information is especially relevant in a particular class, so that this information can be displayed automatically (and in the appropriate format) in the watch window, without you having to expand the various datatypes. The clicks you use to get down to data all add up, especially when dealing with a particularly repetitive debugging task, so this feature can really be helpful. MSDN refers to it here.

The technique seems to be specific for native (presumably C++?) code, but the above link refers to techniques to implement this type of functionality for VB.NET and C#.

So let's look at an example of the problem - as it relates to ObjectARX - and how this technique helps. Here's the default view of two very common ObjectARX datatypes, AcGePoint3d and AcGeMatrix3d:

The default view of the pickpnt variable (an AcGePoint3d) is OK, but all those zeroes do make your eyes hurt. The data displayed for xform (an AcGeMatrix3D) tells you nothing whatsoever of any use - it's the address of the 2D array holding the matrix contents. Here's what you get when you expand these two variables:

You need 5 clicks just to determine that the xform variable is actually the unit matrix, which is way too much.

So what can we do? The autoexp.dat rules format is very simple and is documented in the header of the file itself. I won't reproduce the whole description of the format here, aside from this brief statement:

An AutoExpand rule is a line with the name of a type, an equals sign, and text with replaceable parts in angle brackets. The part in angle brackets names a member of the type and an  optional Watch format specifier.

Thereafter follow some specifics regarding the syntax - for an alternative source of information regarding this, you might also try this CodeGuru article. I'd also recommend looking at the various types listed in the file itself - as usual real-life examples often paint better pictures than abstract descriptions.

So, let's look at what we can do for AcGePoint3d and AcGeMatrix3d. Here are the two entries I added to the autoexp.dat file:

AcGePoint3d =x=<x,g> y=<y,g> z=<z,g>

AcGeMatrix3d =<entry[0][0],g> <entry[0][1],g> <entry[0][2],g> <entry[0][3],g>,<entry[1][0],g> <entry[1][1],g> <entry[1][2],g> <entry[1][3],g>,<entry[2][0],g> <entry[2][1],g> <entry[2][2],g> <entry[2][3],g>,<entry[3][0],g> <entry[3][1],g> <entry[3][2],g> <entry[3][3],g>

The first is fairly simple: it simply tells Visual Studio to display each of the X, Y and Z values of the point, but using the "g" type specifier. Without going into details, this means that they are floating-point values that should be abbreviated to significant digits only. I chose to leave in the labels, x, y & z, to improve readability.

The second is clearly longer - once again it uses the "g" type specifier for each of the 16 entries in the 4x4 matrix. Given the volume of information in the matrix class, I decided to leave out the labels, simply listing the contents with each row separated each by a comma.

Here are the results:

When we have something more meaningful contained in the variables, the descriptions will get longer, of course:

A few tips about the entries in autoexp.dat:

• Don't just add them to the end of the file: they should be part of the [AutoExpand] section, not the [Visualizer] section. I placed mine at line 147.
• It seems that the string itself is limited to 256 characters to the right of the first equals sign (in this day and age - can you imagine?). I was lucky - the AcGeMatrix3d rule eventually came to exactly 256 characters (which is how I found out about the limitation - I had to squeeze out a few redundant spaces for it to work).
• I haven't been able to get multiple lines to display - which would be especially useful for the matrix class, of course. If someone can work it out or find the information on the internet, please post a comment!
• It would also be great to do conditional display of data, especially for union types such as the good old resbuf. While this is not supported directly, you can develop an AddIn to make it work, apparently. See this article for more details.

Debugging using Express Editions

Most of our desktop products support a "plug-in" model of development: you create a DLL (which may or may not be renamed with a number of extensions, such as DBX or ARX) which gets loaded into the calling executable's memory space. This allows the process to share memory with the loaded modules, improving performance over the more archaic IPC (inter-process communication)-based architectures.

[Here begins retracted information...]

While Visual C++ Express Edition supports debugging DLL projects using an external executable, Visual C# Express and Visual VB.NET Express Editions do not. When developing with our products it is extremely common to launch an external executable (for instance AutoCAD's executable, acad.exe), which in turn loads your class module/DLL into its memory space, allowing you to step through the code, watching the contents of variables and all the good stuff that comes with a professional debugging tool.

This essentially means these two tools (Visual C#/VB.NET Express Editions) are crippled when it comes to doing serious development work with a plug-in architecture such as AutoCAD's. Visual C++ Express Edition, on the other hand, apparently allows you to debug DLL projects fully.

[End of retracted information.]

OK - I'm adding some information to this post... the paragraphs above belonged to my initial post, and aren't strictly true. Thanks to Ray Mendoza for raising this issue.

While Visual Basic 2005 (and presumably Visual C#) Express Edition does not make it easy for you to set up and use an external application to debug a class library, it is possible to do. Here's the initial message you get, when you simply try to debug a class library with no addition project set-up performed:

While at first you think "OK, great - let me just add in acad.exe and have it as the startup object", when you come to modify the project settings, there's no easy way to do it. You may be able to add in a separate EXE project to debug, but we don't have or need that for a pre-built EXE such as AutoCAD.

So what can we do? Well, it turns out that all you need to enable debugging in Visual Basic Express Edition is a separate file in the project folder, called "MyProjectName.vbproj.user". This contains the user-specific project settings for the MyProjectName project. In my test I used the default project name for a Class Library, and so the file was called "ClassLibrary1.vbproj.user" and in the same folder as "ClassLibrary1.vbproj". Here are the contents of that file:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    <StartAction>Program</StartAction>
    <StartProgram>C:\Program Files\AutoCAD 2007\acad.exe</StartProgram>
  </PropertyGroup>
</Project>

After creating this file (please make sure the executable path is correct for your system, of course), you will need to reopen the project for it to be detected and read. But once it's found, you can debug your class library project with an external executable, just by hitting F5. Once AutoCAD is loaded, NETLOAD your DLL and away you go.

This file does get created automatically using the Autodesk Managed AppWizard (part of the ObjectARX Wizard - see the previous post for more information).

Now - if someone out there knows an easier way to set this up directly in VB Express, please do let me know, by email or by adding a comment to this post. This is the first time I've used VB Express myself, so there may well be something obvious that I've missed.