digiKam GSoC progress : libface with a ‘k’

This is a progress report after a long time.

Since my last post, my work has involved hours of staring at debug outputs and fixing bugs in libface. I guess I can now say that libface is very usable. Apart from fixing certain segfaults, I also expanded the API a bit.

Till now, you could only give images for detection by passing a filename string to LibFace::detectFaces as:


LibFace libFace = LibFace(DETECT, "."); // Only for Detection
vector<Face> result;
result = libFace.detectFaces(string(argv[i]));

which is from one of the programs in the examples directory. Not a very good idea to load an image from the filesystem if digiKam already has it in memory.

Fortunately now this has been overloaded with :


result = libFace.detectFaces(img->imageData,img->width, img->height, img->widthStep, img->depth, img->nChannels);

Where img is of type IplImage, the OpenCV native image type. This is useful because now digiKam et al won’t have to use IplImage, as the new way is API-agnostic.

Another thing which I modified, does not change the interface, but which is very useful nevertheless, is the management of face ID’s.Every ID has a one-to-one correspondence with a person.

This is how a libface database can be updated:

1. Updating with new faces, whose ID’s are specified by digiKam. These are people whose faces have never been added to the DB before, but digiKam wants to decide the ID’s for them.

2. Updating with faces that have  been given no ID’s. These are totally new people. The Face objects whose ID’s have not been specified will automatically be given the next available ID.

3. Updating with faces that already have matching ID’s in the DB. These are people whose faces have already been added to the DB before. In this case, the new and old face images are averaged and the result is treated as that ID’s face. This way, every person has only one single face image stored for him/her in the libface database.

This was quite complicated to code, because we store the Face images sequentially in a chronological order in the DB.

What I had to do was link the storage index of a face image with it’s ID using one std::map, and then use another map to link each ID to the number of times that ID’s face has been averaged. It’s complicated to explain, you’ll need to look at the code.

This is stored at the end of the DB in this way :


<indexIdMap_0>0</indexIdMap_0>
<indexIdMap_1>1</indexIdMap_1>
<indexIdMap_2>2</indexIdMap_2>
<indexIdMap_3>3</indexIdMap_3>
<indexIdMap_4>4</indexIdMap_4>
<indexIdMap_5>5</indexIdMap_5>
<indexIdMap_6>6</indexIdMap_6>
<indexIdMap_7>7</indexIdMap_7>
<indexIdMap_8>8</indexIdMap_8>
<indexIdMap_9>9</indexIdMap_9>
<indexIdMap_10>10</indexIdMap_10>
<idCountMap_0>5</idCountMap_0>
<idCountMap_1>3</idCountMap_1>
<idCountMap_2>2</idCountMap_2>
<idCountMap_3>5</idCountMap_3>
<idCountMap_4>1</idCountMap_4>
<idCountMap_5>2</idCountMap_5>
<idCountMap_6>3</idCountMap_6>
<idCountMap_7>1</idCountMap_7>
<idCountMap_8>10</idCountMap_8>
<idCountMap_9>1</idCountMap_9>
<idCountMap_10>1</idCountMap_10>

Looking near the end, you can see that the person with ID 8 has 10 occurences.

Also,  the recognise function now also returns the “accuracy” of each recognition. This would very helpful if digiKam wants to present “suggestions” for tagging.

What I’m working on right now is a new KDE-friendly wrapper for libface, named “libkface”, which presents a Qt-only interface to client apps, so that digiKam, Krita, Gwenview, KPhotoAlbum etc can use it very easily if they wish.

Oh, and as much as I want to, I’ll not be coming to Akademy this year because my passport issual has been delayed.

Before ending this post, here’s the obligatory screenshot:

Detected Doggy

I iz detected doggy?? OMG!

22 thoughts on “digiKam GSoC progress : libface with a ‘k’

  1. Wow! Those features would be really cool to have in digikam! Oh, and that’s bad news that you’re missing Akademy. In any case, you’ll know everyone from KDE better when you make it next year :) — but hey, you’ve got the other event to attend! :D

  2. Don’t you think

    result = libFace.detectFaces(img)

    is better compared too:

    result = libFace.detectFaces(img->imageData,img->width, img->height, img->widthStep, img->depth )

    YOu could have another function which takes a QImage or whatever and does necessary conversion too.

    Search for “The Little Manual of API Design” by Jasmin Blanchette, a Qt guy.

  3. Yummy can’t wait to see a screenshot or even better, to test it.
    With non-desctructive change it’s the final touch to make digikam perfect!

  4. @eMerzh Those 2 feature improve digikam a lot. But digikam still has a lot of potential to improve (GUI, editing, …)
    But with those features it will surely be an even more powerful program as before…

  5. Hey, when you talked about that id… isn’t this a perferct candidate for akonadi and the semantic desktop? Association of faces with contacts could be very useful.

    By the way, keep going! Face recognition is an incredibly useful and valuable feature!

  6. Hi, I have installed the libface in my macbook. The face detection is working fine but the face training is running out of memory. I found that when you call libface(“All”, “.”), the program is topping of at 4GB of VM and reporting out of memory error. It is NOT happening with libface(“DETECT”, “.”) which works fine. Is mallocing such huge amount of memory reasonable or is there a memory leak?

    • Hi, there’s no memory leak as far as I know.

      The reason behind this usage is that face recognition algorithm needs to load training data in memory.
      You probably are training with a lot of faces.

      It will work with a small number faces without any problem, but the way to do it is to have a different database for every 100 faces or so. I and Alex will soon modify some things and will also add a HOWTO guide, describing how training/recognition is to be used. Please wait for some time. The example program for recognition isn’t exactly the best way to do things, we will update it.

      • OK, but i was just using one image file that has only one face:

        $ Train

        the problem is when the constructor libface(“ALL”, “.”) is called. It never returns from that call and within that call, it keeps on mallocing till it runs out of memory and exits. Is any database loaded during this call? (I am not sure what happens since i havent looked at the code). As far as I understood it extracts the faces from and adds them to the database. In that case there is only one face. Anyways I will wait for better examples.

    • downloading the new libface library worked :). But is there any documentation of how the database is updated? I first trained it with one face and it says:

      Will train with 1 faces…
      ID 0 assigned to face 0

      Also at the end of the database file i find:
      indexIdMap_0 0 /indexIdMap_0
      idCountMap_0 1 /idCountMap_0

      So far it makes sense. Now I traind with a 2nd different face and it says:

      Will train with 1 faces…
      ID 2 assigned to face 0

      This is weird since the ID assigned should be 1. Also at teh end of the database file i find:

      indexIdMap_0 0 /indexIdMap_0
      indexIdMap_1 0 /indexIdMap_1
      idCountMap_0 1 /idCountMap_0
      idCountMap_0 1 /idCountMap_0

      This is also weird since I was expecting something like:

      indexIdMap_0 0 /indexIdMap_0
      indexIdMap_1 1 /indexIdMap_1
      idCountMap_0 1 /idCountMap_0
      idCountMap_1 1 /idCountMap_1

      Doe it not work across different runs? Or is the library not yet updated. I see you have even mentioned averaging faces and storing only one face for one ID in blog, I wish it worked that way :)

      • Fixed in SVN with revision number 100.

        Explanation of bug: PCA needs more than one face for training. Which is why I add a junk image to the training set if there is only one image to train. Which is why the second face that you were adding was being assigned ID 2 instead of 1.
        The bugfix removes the junk image now when new faces are updated, and now the ID assignment is proper.

        Thanks for the info, we need more people to test and tell us how things work, since I’m concentrating on digiKam at the moment. I and Alex are the only two people working on it.
        In fact, if you can and want to help, do ask Alex for commit rights :)

        About the averaging of faces – this is done to save space. It sort of works. I don’t think it will have as much accuracy as having multiple face vectors of a person, but it works.

        Thanks

  7. Hi do you have any data on the accuracy of face recognition? Also in sn sites, people often upload pics rotated, different intensity, etc. Do you think there is a future of face recognition to become that tolerant?

    • The accuracy of recognition is the accuracy of the Eigenfaces method, can be found everywhere.

      Face recognition can be quite tolerant, Eigenfaces isn’t. After I complete digiKam integration, we’re going to add a new recognition algorithm to libface, which will take local features into account, either Elastic Bunch Graph Matching (EBGM) or AAM (Active Appearance Models). Eigenfaces is only a temporary solution. EBGM-based face recognition is already implemented in a CSU project (search for csuFaceIdEval), but personally I have my eyes on AAM. These are *very* efficient compared to Eigenfaces. But that is something to be done after the GSoC project, owing to a lack of time.

      About rotation, intensity, etc: we will implement some preprocessing of faces before feeding them to the training/recognition, like adjustment based on eye position, histogram equalization, and more.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 44 other followers

%d bloggers like this: