Faces of Facebook 6: the future of face recognition

Read previous: Faces of Facebook 5: real-time personal identification using a smartphone

Winding up the presentation, Alessandro Acquisti lists the current face recognition restrictions, concerns, and depicts the probable future of augmented reality.

Availability of images

I do believe that visual facial searches will become more common. What I mean by this is currently we have text based searches – search engines index text, but they can well index images. And in fact, Google recently started talking about pattern-based image search, although not yet facial images but we are really going to get there. Think about 1995. In 1995 the idea that someone could put a name Alessandro Acquisti in something called a search engine and find all the documents on the Web with the name was unthinkable. Maybe five years from now visual searches and facial searches of the type I am describing will be as common, even though they sound as unthinkable now as text-based searches were ten years ago.

And really, if you see the trend I mentioned earlier with Apple, Facebook, and Google going on a shopping frenzy on startups doing face recognition, you obviously see where the commercial interest is, it is not going to stop.

Cooperative subjects

So the other issue is cooperative subjects. It’s true, our subjects were cooperative. In experiment two we had nice frontal photos, in experiment one we had dating site profile photos where people usually use frontal photos, and sometimes they use the artistic shots, and those ones are faces too. But then again, you can take frontal photos also without being noticed. You can use glasses, there is allegedly the Brazilian police already deploying this for the World Cup in 2014. How long will it take before it can be done on contact lenses? Of course it is impossible to develop now, but five to ten years out contact lenses that tell you in the street what is the last blog written by the person you are just walking by. Is it so unthinkable? I don’t think so.

And then, face recognizers keep getting better, also detecting non-frontal photos. As we were doing these experiments, a new version of PittPatt came out. We started with PittPatt 4.2, we now got PittPatt 5.2; we noticed just in the space of ten months of developing of PittPatt a dramatic increase in the ability of the recognizer to detect and recognize side photos.

Geographical restrictions

Geographical restrictions again – the point is that we were using data on hundreds of thousands, but nevertheless a specific community, while if you want to do it on a nationwide scale, hundreds of millions of people, the computation time increases and the number of false positives increases. This is absolutely true, although cloud computing keeps getting faster and larger. More RAM means larger database you are able to analyze, as well as if the accuracy keeps increasing, the number of false positives keeps decreasing.

By the way, something else that we discovered as we were doing these experiments was that in reality computers aren’t so bad recognizing images – we humans are also very bad, in a sense that when we put a human in a task where the only information they have is two photos of the face of a person, I can tell you it’s really hard for some people to recognize.

We humans are so good at recognizing each other because we use all additional contextual clues such as the body shape of the person, how they walk, maybe how they are dressed. Holistically, we capture this information instantly and we use it to say that is Richard, for instance. But when we only have a shot of a face, our ability of recognizing faces decreases, it is not just computers. And you know what – online social networks are going indeed to provide these additional contextual clues.


So this is why at the start of my talk I was mentioning this idea of Web 2.0 profiles as de facto unregulated Real IDs. And interestingly, the Federal Trade Commission1 some time ago approved the Social Intelligence Corporation, a company which wants to do social media background checks. This suggests, really, this idea that we are starting accepting online social network profiles as the real deal, as the real veritable information about the subject.

Now, this creates cool opportunities for e-commerce. Imagine the first picture I showed from ‘Minority Report’ movie, where, say, the ‘Gap’ company in the street can see you enter the store, connect your information from online because on Facebook you are member of the ‘Gap’ fan club and so forth. And you don’t have to wait until 2054 for this; it will be five, ten years from now or less.

But there are really also ominous implications for privacy. Once again, I am a privacy researcher. I am not in this to make a startup and allow companies to track people. I am here to raise awareness about what is going to happen, what I feel is going to happen.

And I feel that this is very concerning. And the reason is that if you consider the literature on privacy, in fact also the literature on anonymity – it is well known by most of you in this room – we are told to expect that anonymity loves crowds. We are anonymous in a crowd; our privacy is protected in a crowd. But here we have the technology which is truly challenging our perception, as well as our expectations of privacy, in a physical crowd in the street, because a stranger could know your last tweet just by looking at you, and online on a dating site, on Prosper.com and so forth.

We don’t anticipate this because we are not involved to think that strangers can recognize us so easily. Not only that, but I know from my other similar researches which are about behavior economics, that we cannot really anticipate the further additional inferences which become possible once you are re-identified.

The problem is that there is no obviously clear solution to meet this problem that doesn’t come with huge unintended consequences or simply which doesn’t work. The ones which do not work are for instance opt-in or user concern; I find them ineffective because most of the data is already publicly available: Facebook for instance – I was mentioning earlier the case of primary profile photos being by default visible to all. Regulation? But what type of regulation? Do we want to stop researching face recognition? Obviously not, there is so much good that can come out of that.

Finally, one of the two final questions is: what will privacy mean in this future of augmented reality where online and offline data blend? But not only that, if you allow me to extrapolate a little widely, to be a little bold, what else will it mean for our interactions as human beings? What I am talking about is the fact that we have evolved for millions of years to trust our instincts when we meet someone face to face, immediately. Our brain, our biology, our senses tell us something about that person: whether the person is trustable or not, whether we like that person or not, whether that person is young or old, cute or not cute and so forth.

‘Wingman’ app can make this joke possible

‘Wingman’ app can make this joke possible

Will we keep on trusting these instincts that evolved over millions of years? Or will we start trusting more the technology, our contact lenses, our glasses that, as we look at the other person, tell us something about that person?

I will go now with a stereotypical joke (see image). This is the Wingman application of face recognition. A guy and a girl meet at the bar, and of course they are using their iPhones to check each other out. And the guy is thinking: “Mhhh, I am going to see if she is on adultfriendfinder.com”. And the girl is thinking: “Mhhh, I am going to check his credit score”.

Our surveys

So, interestingly, in the survey we ran we asked our subjects both of experiment one and experiment three: “What do you think about this?” Before we had identified them we asked them: “What do you feel about the scenarios where people in the street, complete strangers could know from your face your credit score?” Almost everybody was freaked out. At the same time, almost everyone is revealing online the information which is precisely the information we used for this prediction. So we once again have this paradox. I am not the first one to point it out, the paradox between attitude and behavior. But in a way, I fear that this blending of online and offline data that face recognition, cloud computing, online social networks and statistical re-identification are making possible, is pushing the paradox to its most extreme point, which is the unpredictability of what a stranger may know about you.

Key themes

And that’s why I want to conclude with the themes I started from, because I really hope that the big story here is not the numbers but what is going to happen five – ten years out.

  • Your face is the conduit between online and offline world
  • The emergence of ‘personally predictable’ information
  • The rise of visual, facial searches: search engines allowing people to search for faces like today you search for text
  • Democratization of surveillance, which is not necessarily a good thing
  • Social networks profiles as de facto Real IDs
  • The future of privacy in this augmented reality world


A tale of two futures

The two possible futures

The two possible futures

So let me conclude suggesting that of these two futures (see image) one is definitely creepy, perhaps both are creepy, in a sense that personally I don’t want to live in a world, if I had the choice, where anyone in a bar can know my name. If I go to the bar I like them knowing my name if they are my friends, not everyone. But we don’t know which kind of future we are walking into, so we had better be prepared, and this is why we are doing this kind of research.

Thank you!


1Federal Trade Commission (FTC) is an independent agency of the United States government, whose principal mission is the promotion of consumer protection and the elimination and prevention of what regulators perceive to be harmfully anti-competitive business practices, such as coercive monopoly.

Like This Article? Let Others Know!
Related Articles:

Comments are closed.

Comment via Facebook: