TL;DR: Where's the code?

Hi impatient person, it's here: https://github.com/phlash/faces_in_photos

Having got these PoC scripts working, it's time to stop and reflect: look at the information flows, the user experience, the integration expectations, and build something usable (unlike Picasa, which always was kinda kludgey).

Motivation

Google have killed off Picasa desktop, which Phil has been using for a number of years, in particular because of it's ability to recognise faces and tag / group them. With over 20000 photos, and 200+ people tagged, it's time to get out before losing all that knowledge.

Data escape

Provided the appropriate option is ticked, Picasa will save a .picasa.ini file in each folder it knows about, containing rectangles that locate faces in image files, and where those faces have been tagged, the name tag used. While this preserves the output data, it does not include the trained recognition model Picasa is using to accurately match faces, this is buried in a custom binary database, and unlikely to make any sense outside of the particular recognition engine used, so we need to start again for this...

New recogniser tech

After a bit of rummaging Phil found the excellent face_recognition project from Adam Geitgey: https://github.com/ageitgey/face_recognition and his clear blog series explaining stuff: https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78, fabulous job Adam!

Armed with this code, and after a fair amount of trial & error getting dependencies working (see below), some testing showed that careful tuning of image size and algorithm selection could result in decent face detection through a batch of images without eating vast compute time (I have no GPU to help!). I wrote the first script (face_detect) and left it running for a couple of days.

Sane storage

The detection script writes a machine-readable text file, but at some point I will want to perform more complex queries, so an indexed storage format will be required, preferably one that's usable without a lot more installation / maintenance pain: step forward SQLite :)

While the detector was chewing I wrote an importer for .picasa.ini files (face_picasa), into an SQLite database, then another to import, encode and attempt to group unlabelled faces (face_group) without any training.

Early win!

Having looked at the roughly grouped data, it was apparent that some faces group well, others (especially younger faces) will need training. Then I had a brainwave - I have tagged data from Picasa, on the same images, and it's all in a database with co-ordinates - cue one trivial SQL statement to join up all the image files with similar face location rectangles (Picasa seems to use slightly larger rectangles, so it's an inclusion check) to produce a new table of labelled faces from my new detector - I can train my new tech :)

Sage advice

Having got this far, I ran off the end of the tutorials from Adam and was unsure on how to approach the classification / training stages, so I asked my friend and colleague, GBG's resident data scientist Ian. After my poor problem description, Ian wisely suggested that I probably didn't have enough training data to get a good machine learning model, it may well 'overfit' and have problems, besides that's all hard work with new libraries (Scikit probably), and there are effective simpler means to an end: analysis!

Encoding analysis

Ian suggested I try and improve the basic face matcher that comes in the face_recognition package, by looking at the variance of the elements in a facial encoding (128 of them) across my data set, and weighting towards those with higher variance when doing vector (euclidean) distance measures. He also thought it would speed up the comparison if I reduced the n(n-1) complexity to n by averaging the already labelled/grouped Picasa faces and comparing new data to those averages, rather than every member of a group. Good advice, thanks Ian!

Thus was born face_deviation (that calculates the variance weights), and face_group_by_picasa that uses those weights to compare previously unknown faces with the averaged faces that were known to Picasa.

Database schema's and evolutionary code

Yep - it's a mess: the table names are all messed up (eg: faces, should be picasa_something, groups should be detected_faces, etc.); I need to find a sane way to retain Picasa's original face groupings (which can reference the same face label), so we can experiment with using it's grouping to average faces rather than trying to average all the faces for a label (even if they are over several years of growing up). There is much to be done...

The SQL used to build tag_groups isn't in the code, I typed this by hand:

insert into tag_groups (grow, human) select g._rowid_,f.human from faces as f, groups as g where f.human is not null and f.path=g.path and f.left<g.left and f.top<g.top and f.right>g.right and f.bottom>g.bottom;

gThumb as a viewer

I went looking for a way to visualise outputs, it's tedious picking through log files and firing up eog for sample images. After some initial rummaging at https://alternativeto.net I installed gThumb from the standard distro and reversed out it's data storage for image Catalogs, which are convenient XML files in ~/.local/share/gthumb/Catalogs/<Catalog name>.catalog. The script face_to_gthumb exports labelled image groups as Catalogs, so I can browse through and see how things are going - currently investigating building a plugin/extension for gThumb to bring all the bits together but it seems there is zero (absolutely no) documentation on the source structure and only one 3rd party plugin ever written. I think I'll contact the maintainer. Wish me luck!

I have now started work on the plugin https://github.com/phlash/gthumb-faces, which currently intercepts image loading to mark up faces...

face_recognition install

Adam provides instructions on installing the C library used (dlib) from source here: https://gist.github.com/ageitgey/629d75c1baac34dfa5ca2a1928a7aeaf

Having been through this, I discovered that the python package manager does just as well, so my final install list to get this working was:

# aptitude install python-pip python-setuptools python-numpy libboost-python-dev libjpeg-dev cmake
# pip install dlib
# pip install face_recognition

AshbySoft: Face Recognition (last edited 2018-10-12 20:46:09 by Phlash)