Kean Walmsley


  • About the Author
    Kean on Google+

July 2014

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    








« Reversing the direction of an AutoCAD polyline using .NET | Main | Overriding the grips of an AutoCAD polyline to maintain fillet segments using .NET »

September 19, 2012

Creating a face-recognising security cam with a Raspberry Pi – Part 2

In the first post in this series, we introduced the idea behind the Facecam, a Facebook-enabled security camera running on a Raspberry Pi device.

Over the next three posts we’re going to look in more detail at the system’s components. In this post, we’re going to start by looking at the component – the only one that doesn’t actually run on the Raspberry Pi – that downloads the information from Facebook to build the facial recognition database to be used by the device.

As mentioned last time, I’ve coded this as a .NET application – actually by extending the WinForms sample provided with the Facebook C# SDK – in order to dedicate the Raspberry Pi to performing the security cam function. I’m not actually going to share the code for this application, as it’s really a work in progress. The code itself isn’t very complicated – the Facebook piece is mostly executing FQL, which allows you to use a SQL-like syntax to query the Facebook Graph API – and I’ll be describing the techniques it uses should people want to go ahead and roll their own.

Before looking at the behaviour of the application itself, it’s important to understand more about this “database” that it’s going to help build.

The facial recognition code I’ve adopted is based on work by Shervin Emami and uses the Eigenface algorithm to compare detected faces against a database. To understand more about Eigenface, this article provides a thorough explanation.

Shervin’s OnlineFaceRec sample makes use of OpenCV for this process. I’ve borrowed code from this sample for use on the Raspberry Pi, but today’s application simply builds a file and folder structure that can be used by the OnlineFaceRec application to train the database (and essentially build the facedata.xml file we’ll transfer across to the Raspberry Pi).

There are two main parts of this process: downloading the friend data from Facebook and then parsing that data on disk in order to create a training script to be passed to OnlineFaceRec.

These two functions are reflected by the basic UI of the .NET application:

The simple primary UI for the friend downloader app

Clicking on the first button lets us log in to Facebook and accept the permissions requested by the .NET application. Which reminds me: to create an application that makes use of the Facebook API, you’ll need to register it with Facebook and include the AppId in your code. As you can see below, I called my app “FriendExtractor”:

Facebook login

Once we’re logged in, we see the main dialog which will help us drive the download process:

The dialog we can use to download the friend information

Clicking on the “Populate Friends” button pulls down the list of the current user’s friends from Facebook and populates the list on the left:

The populated friend list

You can see the list has checkboxes associated with each item. Originally the implementation didn’t allow selection of friends to include, which ended in a huge amount of data being downloaded (it turns out I have friends who seem to do nothing other than upload their pictures to Facebook and have 1000+ tagged images up there). Running OnlineFaceRec to train the database with the results ended up in a 400MB facedata.xml file being created. Given the fact the R-Pi has 256MB of RAM< I figured that was a non-starter. :-)

I therefore enabled selection of friends that are actually likely to visit our home – or that I may end up demoing this to at AU – which greatly reduces the amount of probably redundant data that gets downloaded and processed. The list of selected friends gets saved in a simple text file and reloaded automatically.

I also added the ability to start downloading from a particular friend, to allow incremental additions/updates. The first time it’s run, it’s certainly worth starting from the beginning, of course, but you may want to restart from a friend further down the list a later point.

When you select a friend on the left, the “Start Download from Selected” button should come alive. Pushing it does what’s written on the box:

The photos being downloaded

Once completed, you’ll have a folder full of photos of you and your friends. The folder structure contains one level of indirection to use the user ID – as friends can very well have the same name (there’s a teenager named Kean Walmsley who lives in Canada and friended me on Facebook some time ago, for instance) – and inside that folder you’ll find a collection of images:

My photos from Facebook on my local drive

Looking closely, above, you’ll see there are both .jpg and .pgm images. The .jpg files are those downloaded directly from Facebook – nothing surprising there. The .pgm files are created by the .NET application using following algorithm:

  • For each tagged photo of a user
    • Use OpenCV face detection to find the faces in the image
    • Compare the tag location with the faces returned. For the best match…
      • Take a greyscale copy of the cropped area
      • Resize it to 50 x 50
      • Equalize the histogram to get consistent brightness and contrast
      • Save that file to the .pgm format

Regarding the quality of the photos downloaded from Facebook: frankly, it varies. Some photos are poorly tagged, and some are tagged well enough but the face cannot easily be extracted (which sometimes means someone else’s gets picked up in its place). And then some photos simply aren’t well suited to face detection/extraction, so the results end up being just plain strange.

I’ve found that some kind of manual scrubbing process between the download and the script creation/database training helps the quality of the data a great deal. I’ve used a tool called IrfanView to do this: you can search a set of sub-folders for *.pgm files and transfer the results across into “Thumbnail View”, and can then proceed to delete the ugly pics from there. Of course you could do the same thing in Explorer if you had a shell extension that can preview .pgm files, but I wasn’t able to find one. And IrfanView seems to work pretty well.

Here’s a view of some shots of Scott Sheppard and Shaan Hurley, for instance:

Scrubbing Scott and Shaan's Facebook photos

In time I expect a bespoke tool – one that remembers the pics you’ve deleted before, to save you having to repeat that task each time – is the way to go. But that’s for another time.

Once this is done, you should have a fairly clean set of normalised (i.e. of the same size and with similar brightness/contrast) files that can be used to generate a training script for the OnlineFaceRec tool.

The second button on our main dialog runs some code that copies the .pgm files to a separate folder and creates a training script listing them all:

private void btnCreateScript_Click(object sender, EventArgs e)

{

  const string root = "Z:\\fb_photos\\";

  const string destRoot = "Z:\\rp_photos\\";

 

  SortedList<long, long> checkedFriends =

    InfoDialog.LoadCheckedFriends();

 

  // Get the list of PGM files in our source directory

 

  DirectoryInfo di = new DirectoryInfo(root);

  var files = di.GetFiles("*.pgm", SearchOption.AllDirectories);

  StringBuilder sb = new StringBuilder(),

                cur = new StringBuilder();

 

  // We'll maintain an index for the persons and an

  // old user ID, so we can tell when we have changed

  // to a new person's data

 

  int person = 0;

  string ouid = "";

 

  // Maintain a counter of the number of photos for

  // the current person (we only write out 2+ photos,

  // as otherwise they don't train the database),

  // as well as a list of their paths

 

  int photosForCurrent = 0;

  List<string> filesForCurrent = new List<string>();

 

  foreach (FileInfo fi in files)

  {

    var path = fi.DirectoryName;

    var info = path.Split(new[] { '\\' });

    if (info.Length == 4)

    {

      string name = info[2], uid = info[3];

      if (ouid != uid)

      {

        // We have a new user

 

        ouid = uid;

 

        // If the last user had more than 2 photos...

 

        if (photosForCurrent > 2)

        {

          // Let's add them to the training script and

          // copy the files to the destination

 

          person++;

          sb.Append(cur.ToString());

          foreach (string file in filesForCurrent)

          {

            string destFile =

              destRoot +

              Utils.RemoveAccents(file.Substring(root.Length));

            Directory.CreateDirectory(

              Path.GetDirectoryName(destFile)

            );

            File.Copy(file, destFile, true);

          }

        }

 

        // Reset the string, photos and image file info

        // for the current user

 

        cur.Clear();

        photosForCurrent = 0;

        filesForCurrent.Clear();

      }

 

      long id = Int64.Parse(uid);

      if (checkedFriends.ContainsKey(id))

      {

        // Add the information on the current photo to

        // the current user's string

 

        cur.Append(

          String.Format(

            "{0} {1} {2}\r\n",

            person, Utils.RemoveAccents(name),

            "./" +

            Utils.RemoveAccents(fi.FullName).

              Substring(root.Length).Replace('\\', '/')

          )

        );

 

        // Do the same for the photos to copy across to

        // the destination

 

        photosForCurrent++;

        filesForCurrent.Add(fi.FullName);

      }

    }

  }

 

  // Don't forget to tidy up and copy the last user's

  // training string and files across

 

  sb.Append(cur.ToString());

  foreach (string file in filesForCurrent)

  {

    string destFile =

      destRoot + file.Substring(root.Length);

    Directory.CreateDirectory(

      Path.GetDirectoryName(destFile)

    );

    File.Copy(file, destFile, true);

  }

 

  // Finally we write out the string to the training file

 

  File.WriteAllText(

    destRoot + "training.txt", sb.ToString()

  );

}

In case you’re wondering, I’ve kept the two main pieces of the process separate primarily so I can combine my wife’s friends with my own before training the database. :-)

Now it’s a simple matter of running OnlineFaceRec with the training script:

Calling OnlineFaceRec with the trainng script

The output of this tool is pretty interesting: aside from the very important facedata.xml file (the database we’ll use on the Raspberry Pi), we also two files that are created mainly to demonstrate the principle at work.

Firstly we have the average face – which is the absolute average of all the pictures of your friends, and tends to be a bland, slightly androgynous and some might say idealised image of a human face:

out_averageImage.bmp

And seondly we have an image of the Eigenfaces themselves, which map the differences from this average image:

out_eigenfaces.bmp

Neither of these images needs to be transferred across to the Raspberry Pi – their data is captured in the XML database (which stands at around 28 MB for the selected subset of my friends).

In the next post, we’ll look at the implementation of a component that makes use of this XML data to perform facial recognition on images captured by our motion detector.

blog comments powered by Disqus

10 Random Posts