Through the Interface

Blinking lights with a Netduino Plus 2

After blinking LEDs with an Arduino Uno R3 and its younger step-sibling the Digispark, I decided to give the same treatment to their beef-cake second cousin, the Netduino Plus 2.

James Meading had been telling me to take a look at the Netduino for ages, so when I ordered some parts for my (optimistically speaking) forthcoming Kinect for ZX Spectrum integration, I decided to throw in a Netduino Plus 2.

Now the Netduino Plus 2 retails for $60 in the US (and 80CHF here in Switzerland, as the device is assembled in the US), which means it’s at least twice the price of the Arduino Uno R3 (which retails for $30 or just under 30CHF).

But there’s a reason for this: while it’s the same size – it fits perfectly onto my existing Arduino mount, for example – the Netduino Plus 2 comes with a very different set of components and corresponding performance profile.

Aside from some immediately obvious additions – such as a Micro SD slot and an Ethernet port – the Netduino Plus 2 has an STMicro 32-bit microcontroller that runs at a whopping 168 MHz (as does its own younger sibling, the Netduino Go, which has fewer onboard components – making it around $10 cheaper – but implements a plug-and-play architecture for peripherals). This is more than ten times the clock-speed of the Arduino Uno R3 with its 16 MHz ATMega328P. More on this in a little while.

The Netduino’s name comes from the fact it runs a version of the .NET Framework – in this case, the .NET Micro Framework 4.2 – which means you can code for it using Visual C# Express 2010 or (as is my case) Visual Studio 2010 and debug directly from that environment, too. Which is clearly great if you’ve worked with .NET languages in the past, although I personally haven’t found it very painful to adapt to Arduino’s C subset, at least not for smaller projects.

But the .NET Micro Framework clearly comes with some runtime overhead, both in terms of capacity and performance, so some would argue there’s a need for the more advanced hardware profile.

Whether the Arduino or the Netduino is the better choice for a particular project will depend on various factors: toolset familiarity, library availability, cost, OS support (for the development environment), power consumption, network connectivity (which comes at extra cost for the Arduino Uno), etc.

Beyond evaluating the various factors, the question of which to go for ultimately comes down to a religion discussion, as you’ll soon find when you search for opinions. I’m not yet familiar enough with developing for either device to want to join one camp or the other (and I may never – there’s probably a reason I feel so at home in Switzerland, after all ;-).

I’m personally happy to have a number of different tools available to me that I’ll choose between depending on the project at hand (Digispark, Arduino, Netduino Plus 2 and Raspberry Pi – I have a 128M and a 256M Model B, but I’m also reading interesting things about the Model A, particularly with respect to power consumption). I expect that I’ll tend towards using the Netduino when I know for sure the peripherals I need to connect with are supported and when having additional storage and/or Internet connectivity is important (adding Micro SD and Ethernet support will add $45 to an Arduino Uno project, for instance). And when I don’t want the overhead of running a full OS (as is the case for the Raspberry Pi). I dare say there are other dimensions to the decision that will occur to me, over time, but that’s my sense based on my initial playing around.

Anyway, to at least check whether the basics worked on my Netduino device, I went ahead and built and deployed the multiple LED project. This time I went with a comparable but slightly different effect, as it came pre-coded in the sample project.

Blinking lights with a Digispark

I talked about some of the fun I’ve been having with Arduino in this previous post. I eventually added a pushbutton and a potentiometer to that particular circuit, modifying the code to watch for a hardware interrupt – to blink the LEDs only when the button is pressed – but also to apply a variable delay to the blinking (speeding it up or slowing it down depending on the direction in which you turn the knob of the potentiometer).

I’ve now ordered an edge connector that will fit my ZX Spectrum: my next stage for the Arduino-based work is to try to apply signals through it to simulate the pressing of keys on the ZX Spectrum’s membrane keyboard. Once I’ve done that, I know I’ll be able to fire off keystrokes based on external conditions. At which point I’ll just have to see whether my newer 512Mb Raspberry Pi has enough capacity and horsepower to do something meaningful with the data coming from Kinect (via libfreenect, of course, which may make it tricky to get skeleton data… we’ll see if that ends up being a deal-breaker). I’m not sure whether I’ll be able to use the R-Pi directly for all this – it may well make sense to invest in a Gertboard… that looks like a handy add-on to have around, in any case.

So, assuming I can detect basic movements – left, right and jump – I’ll then be able to control Manic Miner and Jet Set Willy (classic platform games) on an original ZX Spectrum using Kinect. At least that’s the plan.

But in the meantime, I received a long-anticipated package through the post, yesterday. It contained three Digisparks and some laser-cut cases for them: the results of funding a project on Kickstarter.

The Digispark is a “teeny tiny, Arduino-compatible, dirt cheap, USB development (and production) platform”. And as you can see here, it really is tiny (trust me – I have fairly average-sized hands ;-).

The chip itself is largely compatible with the Arduino standard and toolset: the driver download comes with the 1.0.3 version of the Arduino IDE (which I was already using for my Arduino projects – replacing it with this version also allows me to target the Digispark subset). The Digispark has an ATtiny85 microcontroller (while my Arduino Uno R3 has an ATmega328P, for comparison), and in general clearly has less capability than the full Arduino.

But for small projects such as my blinking lights – this time with 6 LEDs, as there are fewer pins to work with – it does very well:

You can see here that I’ve used a USB extension cable to avoid connecting the device directly to my PC: when developing with the Digispark you need to unplug the device before each compile and upload – plugging it back in only when this is has been done – which would certainly wear the connector if not using an extension to bear the brunt of this. This extra step makes it a bit fiddly, at first, but you soon get used to it. It’s also worth bearing in mind that given the nature of the Digispark, it’s naturally less robust than a full Arduino, so care should be taken when working with it: a short circuit could damage it irreparably, for instance.

I went ahead and assembled the laser-cut acrylic cases I ordered along with the Digisparks:

And then tried out a small one as part of this project:

The fit was fairly good, although the modest amount of inexpert soldering I performed to attach the pin headers did mean the slot for the USB connector forced the Digispark to be at an angle. I could probably flatten the bottom by cutting the pins and soldering a bit more carefully, I expect.

What I got for Christmas

One of the ulterior motives (although you might really consider it a mere side benefit) for my recent trip back to the UK was to pick up the Christmas presents I’d bought for myself on behalf of my wife and kids. :-)

The first – one of the new Raspberry Pi Model B’s with 512 MB of memory – could have been ordered from Switzerland, but I was a bit frustrated at the mark-up added by the local reseller (the previous one I bought from the UK and had shipped directly here from China… that no longer seems possible now that there’s an official local channel).

Here are my two Raspberry Pi devices side-by-side with their stunning mix & match PetRockBlock cases:

The benefit from this new device is clearly incremental, but I’m hoping it’ll allow my retro gaming console to run PSX games and possibly even the Facecam to perform more reliably (I have doubts it’ll help very much in either area, but we’ll see).

The other part of my present is altogether different, in that it will really let me get involved in something that’s brand new (to me, at least): I ordered an Arduino Experimentation Kit (ARDX) from oomlout.co.uk (love that name :-).

I bought the one from Oomlout largely on the strength of this helpful review, although as it’s a few years old I’m sure there are other alternatives now available.

Here’s the kit from the outside:

And here are the contents of the box:

And, of course, the Arduino Uno R3 microcontroller, itself:

Once I get my head around Arduino projects, I’m going to see about taking it down to an even simpler level: I backed the Digispark project on Kickstarter and so have a couple of these cute little Arduino-compatible chips about to show up through the post. I suspect I have a long way to go before I get to start using them, though.

It’s looking as though 2013 is shaping up to be a fun year!

My new ZX Spectrum

I had a fantastic surprise at the beginning of the week: a friend from work – to whom I’d shown my retro-foolings with the Raspberry Pi – brought a plastic bag to my desk containing an original Sinclair ZX Spectrum and power supply. He had talked about my project to a friend of his back home in Rome, who thought I’d like it (he was otherwise going to throw it away).

The computer was in very good condition externally, at least:

I was impressed by the condition of the keyboard, in particular, as these very quickly became dirty and damaged, back in the day. Perhaps Italy is a less corrosive environment for computers than the UK? :-)

Everything worked when I plugged it in and turned it on (I could hear the distinctive click of the speaker when I hit the keys) but it didn’t display on a modern TV set: it turns out that the Spectrum uses an RF modulator – allowing it to work with TV sets available back in the 80s/90s – but that it’s a simple matter to bypass this component, as the Spectrum generates composite video anyway. There’s even a video on YouTube that steps through the process.

So I dusted off my soldering iron and got cracking with the mod. Opening up the case made it apparent I’d received a fairly early model – which I assume means it's 16 rather than 48K – and you can see a little rust where there was some water damage:

The mod itself went fairly well – I ended up with a not-too-ugly bypass of the modulator unit:

The only problem came when I tried to reconnect the keyboard, as the membrane’s connectors are notoriously fragile (especially after 30 years). At first a few keys worked, but they soon stopped.

Not holding much hope of being able to repair this effectively, I got on the web and found a site that still provides them (and they’re newly manufactured!). A few days after placing my order, I received the replacement membrane through the post.

On to replacing the keyboard membrane…

First I had to get the Spectrum’s metal face-plate off. This took but a little work with my wife’s hair-dryer (you’d be right in thinking this is one of the rare times I’ve had call to use one of these, in recent years ;-) followed by some gentle prying, revealing the rubber keyboard mat:

And under the rubber keyboard we can see the original membrane:

Here are the old and new keyboard membranes, side-by-side:

Replacing the membrane was very easy, then it was a mere matter of replacing the keyboard, face-plate (I didn’t even need additional glue) and screws.

Then it was time to plug it back in and connect it via a standard yellow composite video cable into my TV...

... write some code…

... and see it in action…

So what about playing games? Games for the ZX Spectrum were originally loaded from cassette tape via an audio jack. Many of these games are available online, but in digital format.

A future project is to use one of the tape-related utilities to create a bunch of WAVs or MP3s for the games I want to load and then put them on an old iPod (I have an old iPod Photo that looks sufficiently retro for the task ;-), so that I can relive the experience of loading and playing games. I’ll still probably use the Raspberry Pi to do any serious playing – I’d like to keep my Spectrum in good condition – but it’d be fun to get working.

My friend also tells me that he may well be able to bring me back a Commodore 64 from his next visit to Rome. I can’t wait!

(Grazie mille, Francesco and Andreas! :-)

Virtually done

Over the weekend, from Friday to Sunday, I spent far more hours than I’d have liked finishing off my AU Virtual class for this year, “Secure Your AutoCAD Environment” (I’m increasingly involved in security-related activities for the AutoCAD development organisation). Preparing material for AU Virtual is often pretty tough, whether it’s being recorded in person (I’ve done a couple of those) or being in advance and needing extensively editing.

This year’s format was a bit different, in that it required multiple modules to be recorded separately as opposed to a single, monolithic recording. Better from a consumption perspective, apparently, and probably also a bit better in terms of getting presenters to chunk up their material and record it separately, but it felt to me like it added some overhead during the recording and editing phase.

Anyway, it’s now all done and has been uploaded to the AU website. Hopefully that’s it until I start to get viewers’ questions through.

Now I can start to think about other things, once again. I still have my physical AU classes to prepare for, although thankfully most of the coding is done: I mainly need to update the various projects to make sure they work with the latest releases of products & frameworks that were in Beta when I wrote the code – particularly Visual Studio 2012, ASP.NET MVC 4 and Windows 8 – but it shouldn’t be too arduous.

Speaking of Windows 8… I’ve now installed the RTM build both on my dedicated build machine and inside a Parallels VM, and it looks great. I’m looking forward to using it more as a primary OS.

And speaking of Parallels, the big news on that front is that the Kinect SDK now has a version that supports working in a VM – something I begged Microsoft for over a year ago. Thankfully they listened (it wasn’t just me asking, of course) and delivered v1.6 of the Kinect SDK and its accompanying Developer Toolkit, which allow me to connect my Kinect up to my Mac and run my AutoCAD-related Kinect projects, albeit inside a Windows VM inside Parallels. This should help me do more with Kinect as I now no longer have to switch across to my secondary machine to work with it.

And the final bit of “hot of the press” excitement is around the newly expanded Raspberry Pi Model B, now with 512 MB of RAM. It probably won’t make much difference to the face-recognition security cam project, but may well let me run more advanced games in more complex console emulators. Happy days!

Creating a face-recognising security cam with a Raspberry Pi – Part 4

This is the final part of the series on creating a face-recognising security cam. We started by showing how to get motion detection working, and then followed with an initial overview and then posts on the separate Facebook-downloader tool and the onboard face detection component. In this post, we’ll see how we managed to connect up a USB-powered LED message-board from DreamCheeky.

I was originally inspired to use this device to present the results of the Facecam while having dinner at a friend’s place. He’s a fellow geek – even his welcome mat says so – so we Googled around for a solution and came up with the DreamCheeky, mainly because someone had already created a Linux driver for it.

The device is USB-powered and I’m thankfully able to power both this and the Logitech webcam directly from the Raspberry Pi: I don’t need to resort to a powered USB hub, which is sometimes needed if your webcam needs more juice than the average.

Here are the steps I followed to install and build the necessary components on the Raspberry Pi:

# Create our main folder

 

cd

mkdir led

cd led

 

# Get the libhid source and build it

 

wget "http://alioth.debian.org/frs/download.php/1958/libhid-0.2.16.tar.gz"

tar -xzf *.tar.gz

rm *.gz

cd lib*

./configure && make

sudo make install

 

# We need a symbolic link to the libhid output for later

 

sudo ln -s /usr/local/lib/libhid.so.0 /usr/lib/libhid.so.0

sudo ldconfig

 

# Get the source code for the dcled component

 

wget "http://www.last-outpost.com/~malakai/dcled/dcled-2.0.tgz"

tar -xzf *.tgz

rm *.tgz

cd dc*

 

# Make the dcled component

 

sudo pacman -S make libusb

make

sudo make install

I’m not 100% happy with the above process: for home use it’s OK, but as the required library, libhid is licensed under the GPL license there are serious limitations on how you might want to release this as

We realise that this is a serious impediment. The GPL is a "viral" licence, and you will only be able to use libhid in other GPL projects. We would like to change the licence, but libhid uses the MGE UPS SYSTEMS HID Parser, which is GPL, and thus we cannot. Our solution is to rewrite the HID parser. One of these days. We are also in contact with MGE, trying to convince them to loosen their licence. If the licencing issues are solved, we are likely to re-release libhid under the Artistic Licence.

That said, there is apparently hope that it’ll be possible at some point to swap this module out for an alternative.

Once the needed components are built, it should be a simple matter of calling the dcled command to test whether it displays text properly or not. Here are the usage instructions for dcled:

[pi@alarmpi ~]$ dcled --help

Usage- dcled [opts] [files]

 

    --brightness  -b   How bright, 0-2

    --clock       -c   Show the time

    --clock24h    -C   Show the 24h time

    --bcdclock    -B   Show the time in binary

    --debug       -d   Mostly useless

    --echo        -e   Send copy to stdout

    --help        -h   Show this message

    --message     -m   A single line message to scroll

    --nodev       -n   Don't use the device

    --preamble    -p   Send a graphic before the text.

    --repeat      -r   Keep scrolling forever

    --fastprint   -f   Jump to end of message.

    --speed       -s   General delay in ms

    --test        -t   Output a test pattern

    --font        -g   Select a font

    --fontdir     -G   Select a font directory

 

Available preamble graphics:

 

     1 - dots       - A string of random dots

     2 - static     - Warms up like an old TV

     3 - squiggle   - A squiggly line

     4 - clock24    - Shows the 24 hour time

     5 - clock      - Shows the time

     6 - spiral     - Draws a spiral

     7 - fire       - A nice warm hearth

     8 - bcdclock   - Shows the time in binary

 

Optional fonts:

 

     1 - small      - Very small characters

     2 - sga        - Standard galactic alphabet

     3 - small_inv  - Very small inverted characters

You can easily test the unit, then, by using this command to send a repeatedly scrolling message to the screen:

sudo dcled -r "This is a test"

If you see the following message, then you’ve probably forgotten to run as root via sudo:

hid_force_open failed with return code 6

Couldn't find the device.  Was expecting to find a readable

device that matched vendor 1d34 and product 13.  Is the

device plugged in? Do you have permission?

You can also use the lsusb command to check whether the device has been recognised by the Pi.

Now we have the driver working, it’s a fairly simple matter to create another daemon to look for files in a certain folder and then send the messages they contain to the message-board before deleting them. We’ll use the "-p 7” option to cause the message to be posted with a flame-like pre- & post-amble.

Here’s the C/C++ code I used to implement this:

#include <sys/types.h>

#include <sys/stat.h>

#include <stdio.h>

#include <stdlib.h>

#include <fcntl.h>

#include <errno.h>

#include <unistd.h>

#include <syslog.h>

#include <dirent.h>

#include <string.h>

#include <string>

#include <vector>

#include <iostream>

#include <fstream>

#include <algorithm>

#include <syslog.h>

 

using namespace std;

 

// Input and output folder locations

 

const char * inDir = "/home/pi/faces/out";

 

// Get the list of files in a directory

 

int getdir(string dir, vector<string> &files)

{

  DIR *dp;

  struct dirent *dirp;

 

  if((dp = opendir(dir.c_str())) == NULL)

  {

    char msg[200];

    snprintf(

      msg,

      sizeof(msg)-1,

      "Error(%d) opening %s",

      errno,

      dir.c_str()

    );

    syslog(LOG_INFO, msg);

    return errno;

  }

 

  while ((dirp = readdir(dp)) != NULL)

  {

    files.push_back(string(dirp->d_name));

  }

 

  closedir(dp);

 

  sort(files.begin(), files.end());

 

  return 0;

}

 

int main(void)

{

  // Our process ID and Session ID

 

  pid_t pid, sid;

 

  // Fork off the parent process

 

  pid = fork();

  if (pid < 0)

  {

    exit(EXIT_FAILURE);

  }

 

  // If we got a good PID, then we can exit the parent process

 

  if (pid > 0)

  {

    exit(EXIT_SUCCESS);

  }

 

  // Change the file mode mask

 

  umask(0);

 

  // Open any logs here

 

  openlog("ledmsgd", LOG_PID|LOG_CONS, LOG_USER);

 

  // Create a new SID for the child process

 

  sid = setsid();

  if (sid < 0)

  {

    // Log the failure

 

    syslog(LOG_INFO, "Unable to get SID.");

    closelog();

    exit(EXIT_FAILURE);

  }

 

  // Change the current working directory

 

  if (chdir("/") < 0)

  {

    // Log the failure

 

    syslog(LOG_INFO, "Unable to change working directory.");

    closelog();

    exit(EXIT_FAILURE);

  }

 

  // Close out the standard file descriptors

 

  close(STDIN_FILENO);

  close(STDOUT_FILENO);

  close(STDERR_FILENO);

 

  // Daemon-specific initialization goes here

 

  struct stat st = {0};

 

  if (stat(inDir, &st) == -1)

  {

    mkdir(inDir, 0700);

  }

 

  vector<string> files = vector<string>();

  const char *contents = NULL;

 

  /* The Big Loop */

 

  syslog(LOG_INFO, "main loop begins");

 

  while (1)

  {

    // Get the files in our "in" directory

 

    files.clear();

    getdir(inDir, files);

 

    if ((int)files.size() >= 3)

      contents = files[2].c_str();

    else

      contents = NULL;

 

    if (contents != NULL)

    {

      syslog(LOG_INFO, "found a file");

      syslog(LOG_INFO, contents);

 

      char input[256];

      input[0] = 0;

      strcat(input, inDir);

      strcat(input, "/");

      strcat(input, contents);

 

      char cmd[200];

      snprintf(

        cmd,

        sizeof(cmd)-1,

        "dcled -p 7 %s",

        input

      );

      system(cmd);

 

      remove(input);

    }

    sleep(0.5); /* wait half a second */

  }

 

  closelog();

 

  exit(EXIT_SUCCESS);

}

The simplest way to get this source onto the device is to wget it from this blog. Then it’s a simple matter of building it (again, apologies for the lack of makefile):

wget "http://through-the-interface.typepad.com/files/LedMsgDaemon.cpp"

g++ LedMsgDaemon.cpp -o ledmsgd

Once built you should copy & paste the executable to the appropriate folder:

cp ledmsgd /etc/rc.d

And edit the last line of the /etc/rc.conf file to make sure ledmsgd gets launched along with facerecd on boot:

DAEMONS=(!hwclock syslog-ng network openntpd @netfs @crond @sshd @motion ledmsgd facerecd)

That should be it for getting the last of our components in place. At this stage, you should now have an at least partially functional security webcam, assuming you’ve been able to train and copy across a face database in the form of a facedata.xml file.

For fun, here’s a quick test of my own device at my home’s front door. I hadn’t realised that the human eye (and brain) manages to create the effect of scrolling text from an LED message-board better than when not filtered through a video recording (even when in HD), but then I suppose that saves me the trouble of attempting to protect the innocent (i.e. my Facebook friends, if you can call them that. ;-)

I looked at the debug images that were stored in the ~/faces/debug folder, and saw that in general my face was getting detected appropriately, even if the recognition process clearly still needs tweaking to reduce (eliminate?) the false positives:

Part of the issue clearly stems that we don’t look the same when captured via a security camera (especially when recording a video about the experience :-) as we do when we get tagged in photos on Facebook. It’s very possible the issue goes deeper than that, but I’m going to leave my investigations there, for now.

Retro computing with the Raspberry Pi

I’ve decided to interrupt my series on creating a face-recognising security cam to cover another Raspberry Pi-related topic, this week. I’ll try to finish my series write-up for next Wednesday.

At the recommendation of an old friend from University, I recently devoured the excellent “Ready Player One” by Ernest Cline. Ernest was apparently born about a year before me, so to say that the copious pop-culture references from the 1980s resonated would be a massive under-statement. I just wish I’d come across it sooner (in time to have a shot at solving the Easter Egg and winning a 1981 DeLorean, complete with a sadly non-functioning flux capacitor).

As Don Draper famously said in Mad Men:

Nostalgia - it's delicate, but potent. Teddy told me that in Greek, "nostalgia" literally means "the pain from an old wound." It's a twinge in your heart far more powerful than memory alone.

Ready Player One definitely had me pining to relive some of my early computing experiences – whether programming or playing – and so I decided to have a bit of fun with my second Raspberry Pi device (the first is a little busy… I’ll be telling you more about its exploits over the coming weeks).

I’ve mentioned a few of these experiences before, but over a recent rainy weekend I decided to see if I could track down and build or install emulators for the main computing platforms of my youth onto the Raspberry Pi.

Here’s what I found… (the images of the emulators have been taken on my Mac with the graphics streamed down from the Raspberry Pi and displayed using X11, mainly to make it easier to grab screenshots.)

 

TRS-80 Model III

I can’t remember whether I first programmed on the BBC Micro or the TRS-80, but the TRS-80 Model III was certainly the first computer I had access to at home (well, in my father’s home office). It was also my first exposure to computer games – I remember even now many of the problems in Ghost Town, an early text adventure from Scott Adams (one that you can now play on-line, of all things).

The emulator I ended up building for the TRS-80 was SDLTRS, which actually proved to be straightforward to download and build. I couldn’t track down a posted ROM for the Model III, but I did get one for the Model I, which then allowed me to load LDOS from an emulated floppy-drive:

The dates seem a bit off – I guess I’m using a really recent version of the LDOS ROM in order for it to say copyright 1991, and as for the date being 1912… Y2K issues, anyone?

I’m yet to do anything useful with this emulator – I’m really struggling to remember how to load data from a floppy, for starters – but just seeing it again brings back floods of memories.

 

BBC Microcomputer Model B

This was the first computer I worked with at school and used to spend many, many happy hours fooling around with Logo and BASIC.

The emulator I found for this was BeebEm, which required downloading, extracting and building, in much the same way SDLTRS did.

Again, I don’t yet have anything running inside this emulator (my main memories are of Logo, Chuckie Egg and Elite), but will try to do something fun, at some point.

 

ZX Spectrum

I used to love playing games on the ZX Spectrum (my brother and I shared a Spectrum+, as pictured here). While the TRS-80 and BBC Micro were mostly (for me, at least) about learning, the Spectrum was (almost) all about fun.

We used it mostly for games, but I do remember having lots of fun using it with the Currah Microspeech device to synthesise something approaching human speech.

Photo courtesy of Bill Bertram.

The Spectrum emulator I installed – and this was super simple, no building needed – was Fuse.

Once installed, I went looking for ROMs of games I used to play. There are a lot out there, happily. Here’s a screenshot of Manic Miner in action:

One really good thing about running Manic Miner via X11 is the lack of support for sound (I couldn’t hear the ghastly, discordant 8-bit music that plays once the game has loaded ;-).

 

Atari 2600, Nintendo Entertainment System, Super NES, Nintendo 64, Sega Master System, Sega Mega Drive, Sony Playstation, MAME and more…

 

 

 

 

 

The final touch for my retro computing system was an incredible all-round emulator called RetroArch. To install this and the EmulationStation graphical front-end, I used the very handy RetroPie-Setup script.

I mainly installed this piece as my wife cut her gaming teeth on the Legend of Zelda on the NES, and that seemed the simplest way to get that working. I also wanted to get MAME on the device, as there are a number of classic arcade games I wanted to be able to play, too.

The graphics generated by RetroArch are not piped via X11, so I don’t have any screenshots for that. But the games seem to look and play as well as I remember.

 

Wrap-up

The system looks and works really well. For me it’s just amazing that it’s possible to pack in this kind of capability into such a small (and cheap!) device. I suppose it’s not necessarily the mark of progress – or indeed very surprising – that over the course of 30 years you can shrink the capabilities of all these computers into a credit-card sized device, but it' is nonetheless very cool. Long live nostalgia!

By the way, to finish off this device from an aesthetic perspective, I went ahead and bought a PetRockBlock Pie, a beautiful gaming-focused case from maker of the RetroPie-setup (I’ve used the image from their website – mine is actually an also-very-attractive transparent acrylic version):

This case is ideal for people wanting to wire up old-school controllers (I’m think of raiding my in-laws’ attic to get some NES gamepads to connect up), but for now I’ve gone and bought a wired Xbox 360 controller that comes by default with a USB connector… less retro but also less soldering. ;-) Getting that working was a little tricky, but the xboxdrv driver is really simple to install, and I now have it set up to launch with the classic Q-A-O-P-Space (up, down, left, right, fire) keyboard mapping when playing Spectrum games, and a different configuration when using RetroArch.

I’ll have this with me at AU, in case anyone want to take a look (and maybe even find a screen in the exhibit hall to have a play ;-).

Creating a face-recognising security cam with a Raspberry Pi – Part 3

In the last two posts in this series, we introduced the concept and architecture behind the Facecam, and looked at the desktop-based component to build our face recognition database (facedata.xml).

In this post we’ll take a look at the Raspberry Pi-resident face recognition engine. This component is implemented as a daemon (which is basically the Unix/Linux equivalent of a Windows service, for those unfamiliar with the term) that looks in an “input” folder for images to process and populates an “output” folder with the results of the face recognition process.

In the next post in the series, we’ll look at the implementation of a simple daemon to process the text files in this “output” folder, sending the text to an LED message-board.

But back to today’s topic… the first thing we need to do is to install OpenCV on the Raspberry Pi, for which this post proved very helpful.

# Install OpenCV and its samples

 

sudo pacman -S python2-numpy opencv opencv-samples

 

# Add the pi user to the video group

# (good for testing webcam-related samples)

 

sudo usermod -a -G video pi

At this stage it should be possible for you to browse to and test the OpenCV samples, a step I’m going to skip here.

Next up, we can get the source for the daemon onto the local system and build it. Now at some point I’ll get around to creating a makefile for this, but as it’s only one .cpp file for now I haven’t bothered, although I have put it in a ZIP along with a couple of XML files it depends on.

sudo pacman -S unzip gcc pkg-config

cd

mkdir facerecog

cd facerecog

wget "http://through-the-interface.typepad.com/files/FacRecSrc.zip"

unzip FacRecSrc.zip

g++ `pkg-config opencv --cflags --libs` FaceRecDaemon.cpp -o facerecd

sudo cp facerecd /etc/rc.d

It should now be possible to run the daemon by launching the facerecd command, but first we need to create our input and output directories:

cd

mkdir –p faces/in

mkdir faces/out

mkdir faces/debug

facerecd

To avoid having to launch this manually, we can edit the rc.conf file to have it launch on boot:

sudo nano /etc/rc.conf

Then if you scroll down to the last line in the file, you can make sure both motion and facerecd are there:

DAEMONS=(!hwclock syslog-ng network openntpd @netfs @crond @sshd @motion facerecd)

You’ll also need to edit the motion.conf file, to hook it up to the face detection component:

sudo nano /etc/motion/motion.conf

You need to make sure motion is saving images, and that they get copied to the “input” folder. These are the settings I changed:

output normal

jpeg_filename %C-%v-%q

on_picture_save cp %f /home/pi/faces/in

This breaks the uploader Python script we introduced way back when, but for now I’m just fine with that (it’ll be a trivial change in the code to strip of the frame number from the JPG filename).

Let’s now take a look at he C++ source for our daemon. To get started with writing the daemon, I took the boilerplate code from this article. I then took a big chunk of Shervin Emami’s original face recognition implementation, but stripped out a lot that wasn’t needed (as we’re only reading from the database and don’t need to train it, for instance).

Most of this was using an earlier version of the OpenCV API (which seems to be standard C functions), but I also ended up including some more recent C++ code – OpenCV thankfully provides helpful functions to interoperate between its C and C++ APIs – but that also means the code is less consistent than I’d have liked. At some point I’d like to come back and rework the C-based implementation to use C++ – I’m currently using a horrible mix of strings and char*, just for starters – but I have to find the time (and right now this is working well enough for my purposes). Part of the issue stems from my decision – right or wrong – to code inside Visual Studio, transfer the .cpp across to the Pi via SFTP and then build locally using gcc. If this loop was streamlined – by cross compiling directly on Windows or OSX, or even by editing the C++ code directly on the Pi – then I might be more inclined to take the time to rework the implementation.

Here’s the C++ code from FaceRecDaemon.cpp:

#include <sys/types.h>

#include <sys/stat.h>

#include <stdio.h>

#include <stdlib.h>

#include <fcntl.h>

#include <errno.h>

#include <unistd.h>

#include <dirent.h>

#include <string.h>

#include <string>

#include <vector>

#include <iostream>

#include <fstream>

#include <algorithm>

#include <syslog.h>

#include "cv.h"

#include "cvaux.h"

#include "highgui.h"

#include "opencv2/opencv.hpp"

 

#ifndef BOOL

  #define BOOL bool

#endif

 

using namespace std;

using namespace cv;

 

// Input and output folder locations

 

const char * inDir = "/home/pi/faces/in";

const char * outDir = "/home/pi/faces/out";

const char * debugDir = "/home/pi/faces/debug";

 

// Haar Cascade file, used for Face Detection

 

const char *faceCascadeFilename =

  "/home/pi/facerecog/haarcascade_frontalface_alt.xml";

const char *lblCascadeFilename =

  "/home/pi/facerecog/lbpcascade_frontalface.xml";

const char *faceDataXml = "/home/pi/facerecog/facedata.xml";

 

// Set to 0 if you dont want images of the Eigenvectors

// saved to files (for debugging)

 

int SAVE_EIGENFACE_IMAGES = 1;

 

// You might get better recognition accuracy if you enable this

 

//#define USE_MAHALANOBIS_DISTANCE

 

// Global variables

 

IplImage ** faceImgArr        = 0; // array of face images

CvMat    *  personNumTruthMat = 0; // array of person numbers

 

// Array of person names (indexed by the person number)

 

vector<string> personNames;

 

// Default dimensions for faces in the face recognition database

 

int faceWidth = 120;

int faceHeight = 90;

 

// The number of people in the training set

 

int nPersons                  = 0;

 

// The number of training images

 

int nTrainFaces               = 0;

 

// The number of eigenvalues

 

int nEigens                   = 0;

 

// The average image

 

IplImage * pAvgTrainImg       = 0;

 

// Eigenvectors

 

IplImage ** eigenVectArr      = 0;

 

// Eigenvalues

 

CvMat * eigenValMat           = 0;

 

// Projected training faces

 

CvMat * projectedTrainFaceMat = 0;

 

CvMemStorage* storage         = 0;

static IplImage * small_img   = 0;

double scale = 3;

 

CascadeClassifier faceCascade;

 

#define PAD_FACE 40

#define PAD_FACE_2 80

 

// Function prototypes

 

int getdir(string dir, vector<string> &files);

int  loadTrainingData(CvMat ** pTrainPersonNumMat);

int  findNearestNeighbor(float * projectedTestFace);

int findNearestNeighbor(

  float * projectedTestFace, float *pConfidence

);

bool recognizeFromFile(

  const char *inputFile,

  const char *outputFile,

  CvHaarClassifierCascade *cascade,

  float * projectedTestFace,

  CvMat *trainPersonNumMat

);

IplImage* resizeImage(

  const IplImage *origImg, int newWidth, int newHeight

);

CvRect cropRect(const CvRect rect, int w, int h);

IplImage* cropImage(const IplImage *img, const CvRect region);

CvRect detectFaceInImage(

  IplImage* img, CascadeClassifier &cascade

);

void execute();

 

// Get the list of files in a directory

 

int getdir(string dir, vector<string> &files)

{

  DIR *dp;

  struct dirent *dirp;

 

  if((dp = opendir(dir.c_str())) == NULL)

  {

    char msg[200];

    snprintf(

      msg,

      sizeof(msg)-1,

      "Error(%d) opening %s",

      errno,

      dir.c_str()

    );

    syslog(LOG_INFO, msg);

    return errno;

  }

 

  while ((dirp = readdir(dp)) != NULL)

  {

    files.push_back(string(dirp->d_name));

  }

 

  closedir(dp);

 

  sort(files.begin(), files.end());

 

  return 0;

}

 

// Open the training data from the file 'facedata.xml'

 

int loadTrainingData(CvMat ** pTrainPersonNumMat)

{

  // Create a file-storage interface

 

  CvFileStorage* fileStorage =

    cvOpenFileStorage(

      faceDataXml, 0, CV_STORAGE_READ

    );

  if (!fileStorage)

  {

    syslog(LOG_INFO, "Unable to open training database.");

    return 0;

  }

 

  // Load the names

 

  personNames.clear();  // Make sure it starts as empty

 

  nPersons = cvReadIntByName(fileStorage, 0, "nPersons", 0);

  if (nPersons == 0)

  {

    syslog(LOG_INFO, "No people found in training database.");

    return 0;

  }

 

  // Load each person's name

 

  for (int i=0; i < nPersons; i++)

  {

    string sPersonName;

    char varname[200];

    snprintf(varname, sizeof(varname)-1, "personName_%d", (i+1));

    sPersonName = cvReadStringByName(fileStorage, 0, varname);

    personNames.push_back(sPersonName);

  }

 

  // Load the data

 

  nEigens = cvReadIntByName(fileStorage, 0, "nEigens", 0);

  nTrainFaces = cvReadIntByName(fileStorage, 0, "nTrainFaces", 0);

  *pTrainPersonNumMat =

    (CvMat*)cvReadByName(fileStorage, 0, "trainPersonNumMat", 0);

  eigenValMat =

    (CvMat*)cvReadByName(fileStorage, 0, "eigenValMat", 0);

  projectedTrainFaceMat =

    (CvMat*)cvReadByName(fileStorage, 0, "projectedTrainFaceMat", 0);

  pAvgTrainImg =

    (IplImage*)cvReadByName(fileStorage, 0, "avgTrainImg", 0);

  eigenVectArr =

    (IplImage**)cvAlloc(nTrainFaces*sizeof(IplImage *));

 

  for (int i=0; i < nEigens; i++)

  {

    char varname[200];

    snprintf(varname, sizeof(varname)-1, "eigenVect_%d", i);

    eigenVectArr[i] =

      (IplImage *)cvReadByName(fileStorage, 0, varname, 0);

  }

 

  // release the file-storage interface

 

  cvReleaseFileStorage(&fileStorage);

 

  syslog(LOG_INFO, "Face recognition database loaded.");

 

  return 1;

}

 

// Find the most likely person based on a detection.

// Returns the index, and stores the confidence value

// into pConfidence.

 

int findNearestNeighbor(

  float * projectedTestFace, float *pConfidence

)

{

  double leastDistSq = DBL_MAX;

  int iNearest = 0;

 

  for (int iTrain=0; iTrain < nTrainFaces; iTrain++)

  {

    double distSq = 0;

 

    for (int i=0; i < nEigens; i++)

    {

      float d_i =

        projectedTestFace[i] -

        projectedTrainFaceMat->data.fl[iTrain * nEigens + i];

 

#ifdef USE_MAHALANOBIS_DISTANCE

      // Mahalanobis distance (might give better results

      // than Eucalidean distance)

 

      distSq += d_i*d_i / eigenValMat->data.fl[i];

#else

      // Euclidean distance

 

      distSq += d_i*d_i;

#endif

    }

 

    if (distSq < leastDistSq)

    {

      leastDistSq = distSq;

      iNearest = iTrain;

    }

  }

 

  // Return the confidence level based on the Euclidean distance,

  // so that similar images should give a confidence between

  // 0.5 to 1.0, and very different images should give a

  // confidence between 0.0 to 0.5

 

  *pConfidence =

    1.0f -

    sqrt(

      leastDistSq / (float)(nTrainFaces * nEigens)

    ) / 255.0f;

 

  // Return the found index

 

  return iNearest;

}

 

// Make sure the given rectangle is completely within the given

// image dimensions

 

// Creates a new image copy that is of a desired size

 

IplImage* resizeImage(

  const IplImage *origImg, int newWidth, int newHeight

)

{

  int origWidth = 0;

  int origHeight = 0;

 

  if (origImg != NULL)

  {

    origWidth = origImg->width;

    origHeight = origImg->height;

  }

  if (

    origImg == NULL ||

    newWidth <= 0 || newHeight <= 0 ||

    origWidth <= 0 || origHeight <= 0

  )

  {

    syslog(

      LOG_INFO, "ERROR in resizeImage: Bad desired image size."

    );

    closelog();

    exit(1);

  }

 

  // Scale the image to the new dimensions, even if the aspect

  // ratio will be changed

 

  IplImage *outImg =

    cvCreateImage(

      cvSize(newWidth, newHeight),

      origImg->depth,

      origImg->nChannels

    );

 

  if (newWidth > origImg->width && newHeight > origImg->height)

  {

    // Make the image larger

 

    cvResetImageROI((IplImage*)origImg);

 

    // CV_INTER_CUBIC or CV_INTER_LINEAR is good for enlarging

 

    cvResize(origImg, outImg, CV_INTER_LINEAR);

  }

  else

  {

    // Make the image smaller

 

    cvResetImageROI((IplImage*)origImg);

 

    // CV_INTER_AREA is good for shrinking/decimation,

    // but bad at enlarging

 

    cvResize(origImg, outImg, CV_INTER_AREA);

  }

 

  return outImg;

}

 

CvRect cropRect(const CvRect rectIn, int w, int h)

{

  CvRect roi = CvRect(rectIn);

 

  // Make sure the displayed image is within the viewing dimensions

 

  // Limit the bottom-right from past the image

 

  if (roi.x + roi.width > w)

    roi.width = w - roi.x;

  if (roi.y + roi.height > h)

    roi.height = h - roi.y;

 

  // Limit the top-left from before the image

 

  if (roi.x < 0)

    roi.x = 0;

  if (roi.y < 0)

    roi.y = 0;

 

  // Limit the top-left from after the image

 

  if (roi.x > w-1)

    roi.x = w-1;

  if (roi.y > h-1)

    roi.y = h-1;

 

  // Limit the negative sizes

 

  if (roi.width < 0)

    roi.width = 0;

  if (roi.height < 0)

    roi.height = 0;

 

  // Limit the large sizes

 

  if (roi.width > w)

    roi.width = w - roi.x;

  if (roi.height > h)

    roi.height = h - roi.y;

 

  return roi;

}

 

// Returns a new image, a cropped version of the original

 

IplImage* cropImage(const IplImage *img, const CvRect region)

{

  CvSize size;

  size.height = img->height;

  size.width = img->width;

 

  if (img->depth != IPL_DEPTH_8U)

  {

    syslog(

      LOG_INFO,

      "ERROR in cropImage: unknown image depth given in cropImage()."

     );

    closelog();

    exit(1);

  }

 

  // First create a new (color or greyscale) IPL Image and copy

  // contents of img into it

 

  IplImage *imageTmp =

    cvCreateImage(size, IPL_DEPTH_8U, img->nChannels);

  cvCopy(img, imageTmp, NULL);

 

  // Create a new image of the detected region

  // Set region of interest to that surrounding the face

 

  CvRect checked = cropRect(region, img->width, img->height);

 

  cvSetImageROI(imageTmp, checked);

 

  // Copy region of interest (i.e. face) into a new iplImage

  // (imageRGB) and return it

 

  size.width = checked.width;

  size.height = checked.height;

 

  IplImage *imageRGB =

    cvCreateImage(size, IPL_DEPTH_8U, img->nChannels);

  cvCopy(imageTmp, imageRGB, NULL);  // Copy just the region.

 

  cvReleaseImage(&imageTmp);

 

  return imageRGB;   

}

 

// Perform face detection on the input image, using the given Haar

// cascade classifier. Assumes greyscale for input images.

 

// Returns a rectangle for the detected region in the given image.

 

CvRect detectFaceInImage(

  IplImage* img, CascadeClassifier &cascade

)

{

  CvRect found_face;

 

  if (!small_img)

  {

    small_img =

      cvCreateImage(

        cvSize(

          cvRound(img->width / scale),

          cvRound(img->height / scale)

        ),

        8, 1

      );

  }

 

  cvResize(img, small_img, CV_INTER_LINEAR);

  cvEqualizeHist(small_img, small_img);

 

  Mat imgMat(small_img);

 

  // Detect objects in the small grayscale image.

 

  vector<Rect> objects;

  cascade.detectMultiScale(

    imgMat,

    objects,

    1.1f,

    2,

    CASCADE_FIND_BIGGEST_OBJECT,

    Size(20, 20)

  );

 

  if (objects.size() > 0)

  {

    // Found at least one face

 

    Rect r = (Rect)objects.at(0);

 

    found_face.x = (int)((double)r.x * scale);

    found_face.y = (int)((double)r.y * scale);

    found_face.width = (int)((double)r.width * scale);

    found_face.height = (int)((double)r.height * scale);

  }

  else

  {

    // Couldn't find the face

 

    found_face = cvRect(-1,-1,-1,-1);

  }

 

  return found_face;

}

 

string convert(const string& line)

{

  int len = line.length(); 

  string nLine = "";

 

  if (len)

  { 

    nLine += line[0]; 

 

    for (int i = 1; i < len; ++i)

    {

      if (isupper(line[i]))

        nLine += ' '; 

      nLine += line[i];

    }

  }

  return nLine;

}

 

// Recognize the person in the supplied image

 

bool recognizeFromFile(

  const char *inputFile,

  const char *outputFile,

  const char *debugFile,

  CvHaarClassifierCascade *cascade,

  float * projectedTestFace,

  CvMat *trainPersonNumMat

)

{

  //syslog(LOG_INFO, "recognizeFromFile begins");

 

  // Load the image directly as greyscale

 

  IplImage *checkImg =

    cvLoadImage(inputFile, CV_LOAD_IMAGE_GRAYSCALE);

 

  if (!checkImg)

  {

    char msg[200];

    snprintf(

      msg,

      sizeof(msg)-1,

      "ERROR in recognizeFromFile(): Bad input image file: %s",

      inputFile

    );

    syslog(LOG_INFO, msg);

    return false;

  }

 

  // We'll try to detect the face using LBP

 

  CvRect faceRect = detectFaceInImage(checkImg, faceCascade);

 

  // Make sure a valid face was detected

 

  if (faceRect.width > 0)

  {

    // Get the detected face image

 

    IplImage *faceImg = cropImage(checkImg, faceRect);

 

    // Make sure the image is the same dimensions as the

    // training images

 

    IplImage *sizedImg =

      resizeImage(faceImg, faceWidth, faceHeight);

 

    // Give the image a standard brightness and contrast,

    // in case it was too dark or low contrast

 

    // Create an empty greyscale image

 

    IplImage *equalizedImg =

      cvCreateImage(cvGetSize(sizedImg), 8, 1);

    cvEqualizeHist(sizedImg, equalizedImg);

 

    if (!equalizedImg)

    {

      syslog(

        LOG_INFO,

        "ERROR in recognizeFromFile(): no input image."

      );

      closelog();

      exit(1);

    }

 

    char imageFile[200];

    strcpy(imageFile, debugFile);

    char *dot = strrchr(imageFile, '.');

    *dot = 0;

 

    cvSaveImage(imageFile, equalizedImg);

 

    // If the face rec database has been loaded, then try to

    // recognize the person currently detected

 

    if (nEigens > 0)

    {

      // Project the test image onto the PCA subspace

 

      cvEigenDecomposite(

        equalizedImg,

        nEigens,

        eigenVectArr,

        0, 0,

        pAvgTrainImg,

        projectedTestFace

      );

 

      // Check which person it is most likely to be

 

      float confidence;

      int iNearest =

        findNearestNeighbor(projectedTestFace, &confidence);

 

      int nearest = trainPersonNumMat->data.i[iNearest];

 

      // Write results to the output file

 

      char msg[200];

      snprintf(

        msg,

        sizeof(msg)-1,

        "Recognised %s with a confidence of %f",

        (nearest > 0 ? personNames[nearest-1].c_str() : "nobody"),

        confidence

      );

      syslog(LOG_INFO, msg);

 

      string message =

        (nearest > 0 ?

          convert(personNames[nearest-1]) :

          "Unrecognized"

        );

      std::ofstream o1(outputFile);

      o1 << message.c_str() << std::endl;

 

      std::ofstream o2(debugFile);

      o2 << message.c_str() << std::endl;

    }

 

    // Free the resources used for this frame

 

    cvReleaseImage(&faceImg);

    cvReleaseImage(&sizedImg);

    cvReleaseImage(&equalizedImg);

  }

  cvReleaseImage(&checkImg);

 

  return true;

}

 

int main(void)

{

  // Our process ID and Session ID

 

  pid_t pid, sid;

 

  // Fork off the parent process

 

  pid = fork();

  if (pid < 0)

  {

    exit(EXIT_FAILURE);

  }

 

  // If we got a good PID, then we can exit the parent process

 

  if (pid > 0)

  {

    exit(EXIT_SUCCESS);

  }

 

  // Change the file mode mask

 

  umask(0);

 

  // Open any logs here

 

  openlog("facerecd", LOG_PID|LOG_CONS, LOG_USER);

 

  // Create a new SID for the child process

 

  sid = setsid();

  if (sid < 0)

  {

    // Log the failure

 

    syslog(LOG_INFO, "Unable to get SID.");

    closelog();

    exit(EXIT_FAILURE);

  }

 

  // Change the current working directory

 

  if (chdir("/") < 0)

  {

    // Log the failure

 

    syslog(LOG_INFO, "Unable to change working directory.");

    closelog();

    exit(EXIT_FAILURE);

  }

 

  // Close out the standard file descriptors

 

  close(STDIN_FILENO);

  close(STDOUT_FILENO);

  close(STDERR_FILENO);

 

  // Daemon-specific initialization goes here

 

  struct stat st = {0};

 

  if (stat(inDir, &st) == -1)

  {

    mkdir(inDir, 0700);

  }

  if (stat(outDir, &st) == -1)

  {

    mkdir(outDir, 0700);

  }

  if (stat(debugDir, &st) == -1)

  {

    mkdir(debugDir, 0700);

  }

 

  execute();

 

  exit(EXIT_SUCCESS);

}

 

void execute()

{

  // Load the previously saved training data

 

  CvMat *trainPersonNumMat = 0;

  if (loadTrainingData(&trainPersonNumMat))

  {

    faceWidth = pAvgTrainImg->width;

    faceHeight = pAvgTrainImg->height;

  }

  else

  {

    closelog();

    return;

  }

 

  // Project the test images onto the PCA subspace

 

  float *projectedTestFace =

    (float *)cvAlloc(nEigens*sizeof(float));

 

  // Load the HaarCascade classifier for face detection

 

  faceCascade.load(lblCascadeFilename);

 

  if (faceCascade.empty())

  {

    syslog(

      LOG_INFO,

      "ERROR in main loop: Could not load face detection classifier."

    );

    closelog();

    exit(1);

  }

 

  CvHaarClassifierCascade* cascade =

    (CvHaarClassifierCascade*)cvLoad(faceCascadeFilename, 0, 0, 0);

 

  if (!cascade)

  {

    syslog(

      LOG_INFO,

      "ERROR in main loop: Could not load face detection classifier."

    );

    closelog();

    exit(1);

  }

 

  vector<string> files = vector<string>();

  const char *contents = NULL;

 

  /* The Big Loop */

 

  while (1)

  {

    // Get the files in our "in" directory

 

    files.clear();

    getdir(inDir, files);

 

    if ((int)files.size() >= 3)

      contents = files[2].c_str();

    else

      contents = NULL;

 

    if (contents != NULL)

    {

      char input[256];

      input[0] = 0;

      strcat(input, inDir);

      strcat(input, "/");

      strcat(input, contents);

 

      char *last = strrchr(input, '/');

 

      char output[256];

      output[0] = 0;

      strcat(output, outDir);

      strcat(output, "/");

      strcat(output, last + 1);

      strcat(output, ".txt");

 

      char debug[256];

      debug[0] = 0;

      strcat(debug, debugDir);

      strcat(debug, "/");

      strcat(debug, last + 1);

      strcat(debug, ".txt");

 

      if (

        recognizeFromFile(

          input,

          output,

          debug,

          cascade,

          projectedTestFace,

          trainPersonNumMat

        )

      )

      {

        remove(input);

      }

    }

    sleep(0.5); /* wait half a second */

  }

 

  cvReleaseHaarClassifierCascade(&cascade);

 

  if (storage)

  {

    cvReleaseMemStorage(&storage);

  }

  if (cascade)

  {

    cvReleaseHaarClassifierCascade(&cascade);

  }

 

  cvReleaseImage(&small_img);

 

  closelog();

}

The daemon’s job is to scan the incoming images – saved by the motion detection component and copied into the “input” folder – and find the largest contained face, should one exist. Any detected faces get extracted (and a copy gets saved to our “debug” folder, just so we can see how it’s working) and checked against the face recognition database (once again using the Eigenface algorithm). The results get stored to a text file in the “output” folder, to be displayed on the LED message-board.

The most time-consuming part of this process – by at least an order of magnitude – is the initial face detection. I originally made use of Haar-like face detection, but even with some optimisations I borrowed from an iOS-based OpenCV implementation, I still found it far too slow on the Raspberry Pi (I as looking for <1s results and was finding anything from 5s to 90s, depending on the settings). There’s a lot that can be tweaked to control this – such as increasing the minimum size of the object to be detected – but despite various attempts, it still remained too slow.

At Shervin’s suggestion – he even kindly shared some implementation details from his upcoming book, OpenCV 2 Hotshot: RAW – I ended up making use of LBP face detection, instead. This is apparently only available in the C++ API – hence the mixing of the C and C++ in today’s implementation – and the good news is that it certainly brought the cost of the detection down within acceptable limits.

The bad news is that it generates lots of false positives, which means portions of the image often end up getting detected as faces and the Eigenface algorithm often seems to recognise these faces in the database with a depressingly high confidence level.

I’m sure there’s lots that can be done to improve the quality of the face detection, such as using a mask image. Here’s a thought… if Motion was able to save the information from its “locate” feature to be picked up by the facerecd, that would be even better: we’d only have to search the area of motion in the image.

Philipp Wagner kindly suggested adopting the approach shown in this tutorial, which may well help with performance and/or accuracy. I haven’t yet taken a very close look, as integrating this approach would probably lead to quite fundamental changes in the implementation (and I’ve just got too much else going on, at the moment).

Even though the accuracy leaves a lot to be desired, the results are often very entertaining, but we’ll see more about that, next time. And every so often lightning strikes and the correct person is flagged.

Creating a face-recognising security cam with a Raspberry Pi – Part 2

In the first post in this series, we introduced the idea behind the Facecam, a Facebook-enabled security camera running on a Raspberry Pi device.

Over the next three posts we’re going to look in more detail at the system’s components. In this post, we’re going to start by looking at the component – the only one that doesn’t actually run on the Raspberry Pi – that downloads the information from Facebook to build the facial recognition database to be used by the device.

As mentioned last time, I’ve coded this as a .NET application – actually by extending the WinForms sample provided with the Facebook C# SDK – in order to dedicate the Raspberry Pi to performing the security cam function. I’m not actually going to share the code for this application, as it’s really a work in progress. The code itself isn’t very complicated – the Facebook piece is mostly executing FQL, which allows you to use a SQL-like syntax to query the Facebook Graph API – and I’ll be describing the techniques it uses should people want to go ahead and roll their own.

Before looking at the behaviour of the application itself, it’s important to understand more about this “database” that it’s going to help build.

The facial recognition code I’ve adopted is based on work by Shervin Emami and uses the Eigenface algorithm to compare detected faces against a database. To understand more about Eigenface, this article provides a thorough explanation.

Shervin’s OnlineFaceRec sample makes use of OpenCV for this process. I’ve borrowed code from this sample for use on the Raspberry Pi, but today’s application simply builds a file and folder structure that can be used by the OnlineFaceRec application to train the database (and essentially build the facedata.xml file we’ll transfer across to the Raspberry Pi).

There are two main parts of this process: downloading the friend data from Facebook and then parsing that data on disk in order to create a training script to be passed to OnlineFaceRec.

These two functions are reflected by the basic UI of the .NET application:

Clicking on the first button lets us log in to Facebook and accept the permissions requested by the .NET application. Which reminds me: to create an application that makes use of the Facebook API, you’ll need to register it with Facebook and include the AppId in your code. As you can see below, I called my app “FriendExtractor”:

Once we’re logged in, we see the main dialog which will help us drive the download process:

Clicking on the “Populate Friends” button pulls down the list of the current user’s friends from Facebook and populates the list on the left:

You can see the list has checkboxes associated with each item. Originally the implementation didn’t allow selection of friends to include, which ended in a huge amount of data being downloaded (it turns out I have friends who seem to do nothing other than upload their pictures to Facebook and have 1000+ tagged images up there). Running OnlineFaceRec to train the database with the results ended up in a 400MB facedata.xml file being created. Given the fact the R-Pi has 256MB of RAM< I figured that was a non-starter. :-)

I therefore enabled selection of friends that are actually likely to visit our home – or that I may end up demoing this to at AU – which greatly reduces the amount of probably redundant data that gets downloaded and processed. The list of selected friends gets saved in a simple text file and reloaded automatically.

I also added the ability to start downloading from a particular friend, to allow incremental additions/updates. The first time it’s run, it’s certainly worth starting from the beginning, of course, but you may want to restart from a friend further down the list a later point.

When you select a friend on the left, the “Start Download from Selected” button should come alive. Pushing it does what’s written on the box:

Once completed, you’ll have a folder full of photos of you and your friends. The folder structure contains one level of indirection to use the user ID – as friends can very well have the same name (there’s a teenager named Kean Walmsley who lives in Canada and friended me on Facebook some time ago, for instance) – and inside that folder you’ll find a collection of images:

Looking closely, above, you’ll see there are both .jpg and .pgm images. The .jpg files are those downloaded directly from Facebook – nothing surprising there. The .pgm files are created by the .NET application using following algorithm:

• For each tagged photo of a user
  • Use OpenCV face detection to find the faces in the image
    • For this I used Emgu CV – a .NET wrapper for OpenCV – along with OpenCV’s face detection feature that looks for haar-like features
  • Compare the tag location with the faces returned. For the best match…
    • Take a greyscale copy of the cropped area
    • Resize it to 50 x 50
    • Equalize the histogram to get consistent brightness and contrast
    • Save that file to the .pgm format

Regarding the quality of the photos downloaded from Facebook: frankly, it varies. Some photos are poorly tagged, and some are tagged well enough but the face cannot easily be extracted (which sometimes means someone else’s gets picked up in its place). And then some photos simply aren’t well suited to face detection/extraction, so the results end up being just plain strange.

I’ve found that some kind of manual scrubbing process between the download and the script creation/database training helps the quality of the data a great deal. I’ve used a tool called IrfanView to do this: you can search a set of sub-folders for *.pgm files and transfer the results across into “Thumbnail View”, and can then proceed to delete the ugly pics from there. Of course you could do the same thing in Explorer if you had a shell extension that can preview .pgm files, but I wasn’t able to find one. And IrfanView seems to work pretty well.

Here’s a view of some shots of Scott Sheppard and Shaan Hurley, for instance:

In time I expect a bespoke tool – one that remembers the pics you’ve deleted before, to save you having to repeat that task each time – is the way to go. But that’s for another time.

Once this is done, you should have a fairly clean set of normalised (i.e. of the same size and with similar brightness/contrast) files that can be used to generate a training script for the OnlineFaceRec tool.

The second button on our main dialog runs some code that copies the .pgm files to a separate folder and creates a training script listing them all:

private void btnCreateScript_Click(object sender, EventArgs e)

{

  const string root = "Z:\\fb_photos\\";

  const string destRoot = "Z:\\rp_photos\\";

 

  SortedList<long, long> checkedFriends =

    InfoDialog.LoadCheckedFriends();

 

  // Get the list of PGM files in our source directory

 

  DirectoryInfo di = new DirectoryInfo(root);

  var files = di.GetFiles("*.pgm", SearchOption.AllDirectories);

  StringBuilder sb = new StringBuilder(),

                cur = new StringBuilder();

 

  // We'll maintain an index for the persons and an

  // old user ID, so we can tell when we have changed

  // to a new person's data

 

  int person = 0;

  string ouid = "";

 

  // Maintain a counter of the number of photos for

  // the current person (we only write out 2+ photos,

  // as otherwise they don't train the database),

  // as well as a list of their paths

 

  int photosForCurrent = 0;

  List<string> filesForCurrent = new List<string>();

 

  foreach (FileInfo fi in files)

  {

    var path = fi.DirectoryName;

    var info = path.Split(new[] { '\\' });

    if (info.Length == 4)

    {

      string name = info[2], uid = info[3];

      if (ouid != uid)

      {

        // We have a new user

 

        ouid = uid;

 

        // If the last user had more than 2 photos...

 

        if (photosForCurrent > 2)

        {

          // Let's add them to the training script and

          // copy the files to the destination

 

          person++;

          sb.Append(cur.ToString());

          foreach (string file in filesForCurrent)

          {

            string destFile =

              destRoot +

              Utils.RemoveAccents(file.Substring(root.Length));

            Directory.CreateDirectory(

              Path.GetDirectoryName(destFile)

            );

            File.Copy(file, destFile, true);

          }

        }

 

        // Reset the string, photos and image file info

        // for the current user

 

        cur.Clear();

        photosForCurrent = 0;

        filesForCurrent.Clear();

      }

 

      long id = Int64.Parse(uid);

      if (checkedFriends.ContainsKey(id))

      {

        // Add the information on the current photo to

        // the current user's string

 

        cur.Append(

          String.Format(

            "{0} {1} {2}\r\n",

            person, Utils.RemoveAccents(name),

            "./" +

            Utils.RemoveAccents(fi.FullName).

              Substring(root.Length).Replace('\\', '/')

          )

        );

 

        // Do the same for the photos to copy across to

        // the destination

 

        photosForCurrent++;

        filesForCurrent.Add(fi.FullName);

      }

    }

  }

 

  // Don't forget to tidy up and copy the last user's

  // training string and files across

 

  sb.Append(cur.ToString());

  foreach (string file in filesForCurrent)

  {

    string destFile =

      destRoot + file.Substring(root.Length);

    Directory.CreateDirectory(

      Path.GetDirectoryName(destFile)

    );

    File.Copy(file, destFile, true);

  }

 

  // Finally we write out the string to the training file

 

  File.WriteAllText(

    destRoot + "training.txt", sb.ToString()

  );

}

In case you’re wondering, I’ve kept the two main pieces of the process separate primarily so I can combine my wife’s friends with my own before training the database. :-)

Now it’s a simple matter of running OnlineFaceRec with the training script:

The output of this tool is pretty interesting: aside from the very important facedata.xml file (the database we’ll use on the Raspberry Pi), we also two files that are created mainly to demonstrate the principle at work.

Firstly we have the average face – which is the absolute average of all the pictures of your friends, and tends to be a bland, slightly androgynous and some might say idealised image of a human face:

And seondly we have an image of the Eigenfaces themselves, which map the differences from this average image:

Neither of these images needs to be transferred across to the Raspberry Pi – their data is captured in the XML database (which stands at around 28 MB for the selected subset of my friends).

In the next post, we’ll look at the implementation of a component that makes use of this XML data to perform facial recognition on images captured by our motion detector.

Creating a face-recognising security cam with a Raspberry Pi – Part 1

This series of posts builds upon the mini-series on building a motion-detecting security cam based around the Raspberry Pi. Once you have your motion detecting security cam up and running, you should be able to move on to the next stage: enabling that system to recognize faces that it has been trained against.

My specific project (which I’m calling the Facecam, although I haven’t applied for a trademark ;-) pulls data down from Facebook and uses that to train the face recognition system, but that’s far from being a requirement: it’s also very possible to train the database in other ways.

We’ll go into detail on the various parts over the coming weeks (probably once a week, as I want to continue blogging about other topics, too), but let’s start by introducing the project and the system architecture.

Here’s the elevator pitch: Facecam is a security camera that recognises a resident’s Facebook friends when they come to their front door and allows for tailored communication both to the resident and the visitor.

This would basically mean that instead of the resident receiving a notification email saying “You have a visitor” (as in the case of the motion-detection system we saw previously), the email would read “John Smith visited you”. And John could potentially receive a customized “we’re out” message while standing at the door and even a post on his Facebook wall thanking him for the visit.

A lot of this is clearly contingent on the accuracy of the facial recognition system – something we’ll look at much more closely later on. In case the accuracy isn’t adequate, it would be nice at least to provide some feedback to the visitor, saying “Welcome, John!” (even if it’s actually Megan at the door ;-).

Right, now let’s dive into the system architecture. Here’s a quick diagram of the Facecam system:

Here’s a look at the new components (in addition to the “Motion detection” box we saw previously):

1. A component to download information on the resident’s Facebook friends (.NET desktop application)

This component will build a “friend” database (stored in a file named facedata.xml) to be used at runtime by the security cam. The database will be trained with the friends’ photos as downloaded from Facebook. So the component will need to access – for each friend – both their names and the photos in which they have been tagged (as well as the tag information, of course).

Using the tag information, the component will use OpenCV to attempt to extract the friend’s face (by checking the list of detected faces against the tag location, to get the closest match), and – if it finds it – it will resize, crop and equalize the greyscale intensity of the image before saving it to disk for later use in the database training process.

This is ultimately an offline, CPU-intensive operation – we will simply need the database transferred across to the Raspberry Pi from time to time – so the approach we’ll take is to create a .NET desktop application that performs this task on a higher-powered device.

2. A component to analyse frames saved by the motion-detection system, checking for friends (face recognition)

This is the guts of the system: the runtime component that will again use OpenCV to process images captured from the webcam, detect faces and then check the results against the facial recognition database on the Raspberry Pi. This needs to be pretty efficient – as the feedback should be in close to real-time – which may be a challenge: given the relatively low power of the Raspberry Pi CPU, we’ll have to jump through some interesting hoops to get this working well enough.

3. A component to provide feedback to the visitor (LED messaging)

If we’re going to tell the visitor that they have been recognised (or in some way warn unwanted visitors that their photo has been sent to the property’s residents), then we’re going to need some kind of screen. The Raspberry Pi has both HDMI and component video outputs, but I’ve decided to go the way of a USB-powered LED message-board. More on this in due course.

Right, that’s it for this introductory post. We’ll look in more detail at each of these components over the coming weeks, starting with the .NET desktop application to download friend information.