Now things are getting interesting. :-)
It took me some effort to get this working, but looking at the results I think it was worth it. As mentioned in the last post, I’ve been trying to get a live feed from the Kinect sensor to display dynamically inside AutoCAD. This post shows how to do that.
A huge thanks to Boris Scheiman for helping me get to this point: he not only extended his nKinect implementation to generate the point cloud data I asked for, he sent a number of emails explaining (and even providing) the various modules needed to get nKinect up and running.
Before we get into the implementation, I should add some information about general Kinect development. In addition to the four options for developing Kinect apps mentioned in the last post, there’s a very important one I omitted: OpenNI (NI standing for Natural Interaction). This was founded – at least in part – by PrimeSense, the company who provides the 3D-sensing technology to Microsoft for their Kinect sensors. PrimeSense co-founded OpenNI back in November, when they released their own drivers that could be used with the Kinect. Boris chose to move the nKinect implementation across to OpenNI (although the Codeplex site says otherwise, hence the confusion in my last post). As such, OpenNI is the closest thing to an “official” implementation (until Microsoft’s Kinect SDK is available, that is).
It took me much, much longer than I’d hoped to get the OpenNI implementation up and running: there are three main components to install for nKinect to work, and it’s essential to get matching versions of each. The versions I ended up using were suggested and provided by Boris: I couldn’t have managed without him as they no longer appear to be available on the web.
Let’s talk a bit about these various components and what they’re for.
- OpenNI SDK
- SensorKinect
- PrimeSensor modules for OpenNI
- As far as I can tell this adds an additional Kinect-specific OpenNI-compliant driver, as well as some additional files to make it easier to get OpenNI and NITE working with the Kinect
- I installed the SensorKinect-Win32-5.0.0.exe – this has not been updated since early January, and so needs old, unstable builds of OpenNI and NITE
- PrimeSensor modules for OpenNI
- NITE Middleware
- Provides higher-level capabilities such as user identification, feature and gesture detection
- Both stable and unstable builds of the SDK are available
- At the time of writing 1.3.1.3 (stable) and 1.3.1.4 (unstable) are available
- I’m using 1.3.0.17 (also unstable – I think), as that’s the one needed by SensorKinect
- You need to use this license key when installing the SDK: “0KOIk2JeIBYClPWVnMoRKn5cdY4=”
Once all these components have been installed, you need to copy some XML files installed with SensorKinect into the folders for OpenNI and NITE. The license key needs to be entered in one of these.
Then there’s nKinect itself, which can be downloaded and placed anywhere on your system. It has a dependency on OpenCV, so I also needed to get the right version of that (2.2), copying across a subset of its DLLs (as reported by Dependency Walker after loading the main nKinect.dll). There’s also an OpenNI.xml configuration file that needs to be in a Data folder beneath your .DLL module’s location.
nKinect abstracts away a lot of the hard work needed to deal with Kinect input: you get a couple of callbacks for when the RGB and depth data is updated, so you can update your own UI. It looks, in fact, very similar to functionality that was presented yesterday at the MIX 11 Day 2 keynote (starting about 1 hour 40 mins in). I do expect I’ll be porting across to that – as it clearly gets rid of a large amount of configuration trouble, as you can see from the above steps – but nKinect has certainly allowed me to get up and running in advance of that being available.
So what does the nKinect-based code look like?
Here’s my C# implementation, which has a DrawJig taking the point data provided by nKinect, displaying it dynamically to the user until a point is selected. Much of this code is actually used to bring in the point cloud from a TXT file via LAS to PCG (using txt2las and the implementation show in the Browse Photosynth Plugin of the Month).
using Autodesk.AutoCAD.ApplicationServices;
using Autodesk.AutoCAD.DatabaseServices;
using Autodesk.AutoCAD.EditorInput;
using Autodesk.AutoCAD.Geometry;
using Autodesk.AutoCAD.GraphicsInterface;
using Autodesk.AutoCAD.Runtime;
using System.Runtime.InteropServices;
using System.Collections.Generic;
using System.Windows.Media;
using System.Diagnostics;
using System.Reflection;
using System.IO;
using System;
using NKinect;
namespace KinectIntegration
{
public class KinectJig : DrawJig
{
// We need our nKinect sensor
private Sensor _kinect = null;
// A list of points to be displayed
private Point3dCollection _points;
// A list of points captured by the sensor
// (for eventual export)
private List<ColorVector3> _vecs;
// The constructor takes a Sensor argument
public KinectJig(Sensor kinect)
{
_kinect = kinect;
_points = new Point3dCollection();
}
protected override SamplerStatus Sampler(JigPrompts prompts)
{
// Se don't really need a point, but we do need some
// user input event to allow us to loop, processing
// for the Kinect input
PromptPointResult ppr =
prompts.AcquirePoint("\nClick to capture: ");
if (ppr.Status == PromptStatus.OK)
{
// Generate a point cloud via nKinect
_vecs = _kinect.GeneratePointCloud();
// Extract the points for display in the jig
// (note we only take 1 in 5)
_points.Clear();
for (int i = 0; i < _vecs.Count; i += 5)
{
ColorVector3 vec = _vecs[i];
_points.Add(
new Point3d(vec.X, vec.Y, vec.Z)
);
}
return SamplerStatus.OK;
}
return SamplerStatus.Cancel;
}
protected override bool WorldDraw(WorldDraw draw)
{
// This simply draws our points
draw.Geometry.Polypoint(_points, null, null);
return true;
}
public void ExportPointCloud(string filename)
{
if (_vecs.Count > 0)
{
using (StreamWriter sw = new StreamWriter(filename))
{
// For each pixel, write a line to the text file:
// X, Y, Z, R, G, B
foreach (ColorVector3 pt in _vecs)
{
sw.WriteLine(
"{0}, {1}, {2}, {3}, {4}, {5}",
pt.X, pt.Y, pt.Z, pt.R, pt.G, pt.B
);
}
}
}
}
}
public class Commands
{
[CommandMethod("ADNPLUGINS", "KINECT", CommandFlags.Modal)]
public static void ImportFromKinect()
{
Document doc =
Autodesk.AutoCAD.ApplicationServices.
Application.DocumentManager.MdiActiveDocument;
Editor ed = doc.Editor;
Sensor kinect = null;
KinectJig kj = null;
PromptResult pr = null;
try
{
kinect = new Sensor(s => { }, r => { }, d => { });
kinect.Start();
kj = new KinectJig(kinect);
pr = ed.Drag(kj);
}
catch { }
finally
{
if (kinect != null)
{
kinect.Stop();
kinect.Dispose();
}
}
if (pr.Status == PromptStatus.OK)
{
// We'll store most local files in the temp folder.
// We get a temp filename, delete the file and
// use the name for our folder
string localPath = Path.GetTempFileName();
File.Delete(localPath);
Directory.CreateDirectory(localPath);
localPath += "\\";
// Paths for our temporary files
string txtPath = localPath + "points.txt";
string lasPath = localPath + "points.las";
// Our PCG file will be stored under My Documents
string outputPath =
Environment.GetFolderPath(
Environment.SpecialFolder.MyDocuments
) + "\\Kinect Point Clouds\\";
if (!Directory.Exists(outputPath))
Directory.CreateDirectory(outputPath);
// We'll use the title as a base filename for the PCG,
// but will use an incremented integer to get an unused
// filename
int cnt = 0;
string pcgPath;
do
{
pcgPath =
outputPath + "Kinect" +
(cnt == 0 ? "" : cnt.ToString()) + ".pcg";
cnt++;
}
while (File.Exists(pcgPath));
// The path to the txt2las tool will be the same as the
// executing assembly (our DLL)
string exePath =
Path.GetDirectoryName(
Assembly.GetExecutingAssembly().Location
) + "\\";
if (!File.Exists(exePath + "txt2las.exe"))
{
ed.WriteMessage(
"\nCould not find the txt2las tool: please make sure " +
"it is in the same folder as the application DLL."
);
return;
}
// Export our point cloud from the jig
ed.WriteMessage(
"\nSaving TXT file of the captured points.\n"
);
kj.ExportPointCloud(txtPath);
// Use the txt2las utility to create a .LAS
// file from our text file
ed.WriteMessage(
"\nCreating a LAS from the TXt file.\n"
);
ProcessStartInfo psi =
new ProcessStartInfo(
exePath + "txt2las",
"-i \"" + txtPath +
"\" -o \"" + lasPath +
"\" -parse xyzRGB"
);
psi.CreateNoWindow = false;
psi.WindowStyle = ProcessWindowStyle.Hidden;
// Wait up to 20 seconds for the process to exit
try
{
using (Process p = Process.Start(psi))
{
p.WaitForExit();
}
}
catch
{ }
// If there's a problem, we return
if (!File.Exists(lasPath))
{
ed.WriteMessage(
"\nError creating LAS file."
);
return;
}
File.Delete(txtPath);
ed.WriteMessage(
"Indexing the LAS and attaching the PCG.\n"
);
// Index the .LAS file, creating a .PCG
string lasLisp = lasPath.Replace('\\', '/'),
pcgLisp = pcgPath.Replace('\\', '/');
doc.SendStringToExecute(
"(command \"_.POINTCLOUDINDEX\" \"" +
lasLisp + "\" \"" +
pcgLisp + "\")(princ) ",
false, false, false
);
// Attach the .PCG file
doc.SendStringToExecute(
"_.WAITFORFILE \"" +
pcgLisp + "\" \"" +
lasLisp + "\" " +
"(command \"_.-POINTCLOUDATTACH\" \"" +
pcgLisp +
"\" \"0,0\" \"1\" \"0\")(princ) ",
false, false, false
);
doc.SendStringToExecute(
"_.-VISUALSTYLES _C _Conceptual _.ZOOM _E ",
false, false, false
);
//Cleanup();
}
}
// Return whether a file is accessible
private bool IsFileAccessible(string filename)
{
// If the file can be opened for exclusive access it means
// the file is accesible
try
{
FileStream fs =
File.Open(
filename, FileMode.Open,
FileAccess.Read, FileShare.None
);
using (fs)
{
return true;
}
}
catch (IOException)
{
return false;
}
}
// A command which waits for a particular PCG file to exist
[CommandMethod(
"ADNPLUGINS", "WAITFORFILE", CommandFlags.NoHistory
)]
public void WaitForFileToExist()
{
Document doc =
Application.DocumentManager.MdiActiveDocument;
Database db = doc.Database;
Editor ed = doc.Editor;
HostApplicationServices ha =
HostApplicationServices.Current;
PromptResult pr = ed.GetString("Enter path to PCG: ");
if (pr.Status != PromptStatus.OK)
return;
string pcgPath = pr.StringResult.Replace('/', '\\');
pr = ed.GetString("Enter path to LAS: ");
if (pr.Status != PromptStatus.OK)
return;
string lasPath = pr.StringResult.Replace('/', '\\');
ed.WriteMessage(
"\nWaiting for PCG creation to complete...\n"
);
// Check the write time for the PCG file...
// if it hasn't been written to for at least half a second,
// then we try to use a file lock to see whether the file
// is accessible or not
const int ticks = 50;
TimeSpan diff;
bool cancelled = false;
// First loop is to see when writing has stopped
// (better than always throwing exceptions)
while (true)
{
if (File.Exists(pcgPath))
{
DateTime dt = File.GetLastWriteTime(pcgPath);
diff = DateTime.Now - dt;
if (diff.Ticks > ticks)
break;
}
System.Windows.Forms.Application.DoEvents();
}
// Second loop will wait until file is finally accessible
// (by calling a function that requests an exclusive lock)
if (!cancelled)
{
int inacc = 0;
while (true)
{
if (IsFileAccessible(pcgPath))
break;
else
inacc++;
System.Windows.Forms.Application.DoEvents();
}
ed.WriteMessage("\nFile inaccessible {0} times.", inacc);
try
{
CleanupTmpFiles(lasPath);
}
catch
{ }
}
}
internal static void CleanupTmpFiles(string txtPath)
{
if (File.Exists(txtPath))
File.Delete(txtPath);
Directory.Delete(
Path.GetDirectoryName(txtPath)
);
}
}
}
When running the code the user should see the jigged point-cloud. It’s possible to zoom, pan and orbit to get the point cloud feed in the view (this can be a little tricky, admittedly).
Selecting a point (an arbitrary one – that’s just allowing us to update the data being displayed, we don’t actually care about the point itself) will cause the full import process to occur. The data imported is that displayed most recently inside AutoCAD.
And just to show it’s a full 3D point cloud – albeit captured from a single position – here’s another angle:
Now I need to start looking at interpreting gestures, to see what can be done to supplement AutoCAD’s existing user interface features. Fun fun fun! :-)
Update:
Thanks to prompting from Guillermo Bellmann, I managed to download and install the following module versions before they were superseded (all this stuff is really in the Wild West of coding ;-).
-
OpenNI 1.1.0.39 (unstable)
-
SensorKinect 5.0.1.32
-
NITE 1.3.1.3 (unstable)
I had to make some minor modifications to nKinect to get it working, mainly to incorporate the updated headers and libraries from OpenNI and NITE. I can now confirm that it’s possible to make the above code work with these versions.
On a side note, in trying to troubleshoot a few stumbling blocks, I stumbled across these useful posts, which help explain the history and functionality of these components.
