Faces of Facebook: face recognition technologies

Alessandro Acquisti Professor Alessandro Acquisti from Carnegie Mellon University takes the floor at Black Hat USA to speak on the role of Facebook and social media in face recognition research and advancement.

Thanks everybody for being here, it’s always a great pleasure to be at Black Hat, and in this case I would like to mention my co-authors Ralph Gross and Fred Stutzman. This talk is about a few things: face recognition for sure, that’s where we start from; but face recognition is just a start. What we were interested in studying was the not so distant consequences and implications of the convergence of a number of technologies.

One of the other technologies are online social networks and, combined with a few more ingredients that you will see through my presentation, the story we are trying to tell is the story of augmented reality, of a blending of online and offline data, which I find inevitable, and which raises pretty deep and concerning privacy questions.

Targeted advertising in ‘Minority Report’ movie mall scene

Targeted advertising in ‘Minority Report’ movie mall scene

So this is what the talk is about, but I will start not from the talk itself but from the future. This is the movie that we have all seen – ‘Minority Report’. Tom Cruise is walking around a future shopping mall (see image). I believe that in the movie version of Philip K. Dick’s story, it was 2054. And somehow, the advertising is changed and adapted in real time based on the combination of technologies like face recognition and iris scan. So customers walking in the mall are recognized and advertising is targeted. The same way now online advertising is targeted based on your online behavior.

‘Minority Report’ iris scan scene

‘Minority Report’ iris scan scene

But there is also another side to this future in the movie, the creepier side, where iris scan is used for surveillance: identifying and detecting people. So keep in mind these two images, these two potential futures we are walking into. They are not mutually exclusive. Keep them in mind and we will go back to this, but in the meanwhile I will jump back into the past. I feel the best summary of the state of facial recognition today was given actually in 2010 by another Black Hat and Defcon speaker Joshua Marpet, he said: “Facial recognition sucks. But it’s getting better”. And I would strongly subscribe to these words. Computers are still way worse than humans in recognizing faces, but they keep getting better.

LED hat – confusion for face recognizer

LED hat – confusion for face recognizer

They can still be fooled. At a Defcon presentation, I believe last year or two years ago, a pretty smart idea was about putting some LEDs on a hat, which was sufficient to confuse a face recognizer (see image). This was part of the badge competition at Defcon. But they said: “Granted that face recognizer still somewhat sucks, also granted that the first derivative is clear – they keep getting better and better”.

Research in this area has been going on for more than forty years. And they keep getting better so much that they start being used in real applications – not just in security, but in fact also more recently in the Web 2.0 applications.

And in fact, if you see what has been happening in the past few months or at most a couple of years, pretty much every startup doing good work in face recognition is being acquired either by Google, or by Facebook, or by Apple. Google has acquired Neven Vision a few years ago, and then more recently Riya, and then even more recently PittPatt. PittPatt is actually interesting because it is the software that we used for our experiments. It was developed by researches at CMU (Carnegie Mellon University) where I am from. And then pretty much after our experiments were done, a few weeks ago we heard the news that PittPatt had just been acquired by Google. Similar story with Polar Rose acquired by Apple, and Facebook didn’t acquire Face.com but licensed its technology and started using it in automated tagging and so forth. So there is obviously a very significant commercial interest, as well as governmental interest in face recognition.

So what are we doing that I think is potentially different? Well, I am a researcher, I am a privacy researcher, I am not a face recognition researcher, although Ralph Gross, my co-author, is; so we took this view of extrapolating five-ten years out to what will be possible to do with face recognition: online social network, cloud computing, statistical re-identification, and data mining. So in a way, the mix, the mashup of these technologies is what can be created.

1. Continuing improvements in face recognizer’s accuracy

First of all, as I have already mentioned a couple of times, face recognition keeps improving. The FERET program, Department of Defense program to create metrics for the accuracy and performance of face recognizers, clearly shows that even just in close to ten years between 1997 and 2006, there were improvements of around two orders of magnitude.

In 1997, best face recognizer in FERET program scored error rate of 0.54
By 2006, error rate was down to 0.01

If you see the data now, although I do not know if the FERET program is still running, but if you see the performance of face recognizers now, it has gone up dramatically.

2.Increasing public self-disclosures through online social neworks

Also, something which we didn’t have that much in 2006, and definitely we didn’t have in 1997 was the incredible amount of personal information that you could find about strangers online, and in particular – photos.

In 2000 – this is the statistics I found in ‘Science’, the peer-reviewed journal – there were about 100 billion photos shot worldwide. The researchers who estimated this number did it in a very smart way: they found data on the worldwide sales of photo cameras, and then they estimated from that the number of shots that could have been made that year. Only a negligible portion of those shots made it online. But in fact, ten years later, only Facebook users, in just a month, were uploading 2.5 billion photos. Many of those photos were of course depicting faces, and many of those faces could be connected to an identity, because either they are primary profile photos or they are tagged with someone’s name. In fact, many Facebook users use their real first and last name when they create an account on the network.

3.Statistical re-identification

Other technologies that we are considering is this big mashup which creates the problem that this study is about – statistical re-identification. If I had to give a birth date to it, it would be probably 1997, when Latanya Sweeney, at the time she was at MIT (Massachusetts Institute of Technology) and then she moved on to Carnegie Mellon as a professor, discovered something that if you look back at now – it seems obvious, but truly at the time she came out with this idea, it was revolutionary.

She took people’s date of birth, which is personal information but not unique personal information. She then took people’s gender, which again is personal information but not personally identifiable information. And she took people’s zip code of the address where they live, which, once again, is personal but not unique. And combining the three, she found that 87% of America’s population were uniquely identifiable by the combination of these three pieces of data. If you imagine this as a Venn diagram1, you find out that many of us are indeed uniquely identifiable by these three pieces of data.

This is an example of the power of statistical re-identification, to start from data which is not particularly sensitive and create something which is potentially more sensitive, such as your unique identity.

The Netflix prize research by Narayanan and Shmatikov was another very good example. The scholars took data from the Netflix prize, which had been anonymized, ratings of movies, they found identity of the rater, and combined it and compared it to data from the IMDB (Internet Movie Database) and showed that they could statistically re-identify a statistically significant proportion of the members of the Netflix set.

And then of course there was our own research two years ago when we presented a study where we showed that we could predict social security numbers from public data, and in fact for instance from online social network profiles. And the story here was that we could start from something personal but not so sensitive – your date of birth and your state of birth, combine it with also publicly available data such as the Death Master File, which is a database of social security numbers of people who are dead. And by combining the two and doing some statistical analyses, we could end up predicting the SSN not of the dead people but of people who were alive.

4.Cloud computing

Another technology is cloud computing. Cloud computing allows cheaply and efficiently running many, many face comparisons in just a few seconds, otherwise that would not be possible with normal computers.

5.Ubiquitous computing

The idea of ubiquitous computing is that I can just take my smartphone and connect it to the Internet, and although my smartphone does not have the processing power to do 500 million face comparisons in seconds, something up there in the cloud can, and I just need to connect to it to run face recognition in real time in the street.

So this is what we are talking about: combining all these technologies, and in particular face recognition and publicly available online social network data, for the purpose of large-scale, automated, peer-based individual re-identification both online and offline; and individual informational inference, the inference of additional information about these individuals, potentially sensitive data.

Read next: Faces of Facebook 2: facial taxonomy and statistical re-identification

1Venn diagram, or set diagram, is a diagram that shows all possible logical relations between a finite collection of sets (aggregation of things). This type of diagrams is used to illustrate simple set relationships in probability, logic, statistics, linguistics and computer science.

Like This Article? Let Others Know!
Related Articles:

Comments are closed.

Comment via Facebook: