Outlined here are a number of experiments conducted by Marti Motoyama and his colleagues in order to figure out the demographic patterns for CAPTCHA solvers.By looking at the labor demographics, we can better understand the cogs that operate within these CAPTCHA-solving machines; perhaps new CAPTCHAs can be developed that capitalize on this labor demographic knowledge. We developed two CAPTCHAs, I only have time to show you one. The CAPTCHA asked the user to translate the words zero through nine into their corresponding numeric values (see right-hand image).
So, for example, we asked the user to convert the spelled out word “seven” to the number 7, and we did this for over 20 languages. We submitted 222 unique CAPTCHAs per language to each of the six services still operating when we conducted this experiment. So, AG corresponds to Antigate, PC – BeatCaptchas, BY – BypassCaptcha, CB – CaptchaBot, DC – Decaptcher, IT – ImageToText.The table shown on the screen shows the percentage of correct responses we received for each language per service (see left-hand image). English and Chinese are well represented across most services; otherwise the services exhibit different affinities for different languages. For example, Antigate and CaptchaBot have a strong Russian presence, while BeatCaptchas has a strong Tamil and Portuguese speaking populations. And this is very logical, as the regions where they speak these languages correspond to low-cost labor markets.
We see this very oddness with ImageToText: they really have a remarkable ability to translate this CAPTCHA where all the other services really seem to fail. Not only that, ImageToText was even able to solve CAPTCHAs from the synthetic language that we incorporated into this experiment, Klingon. The main takeaway from this slide is that the low-cost labor markets – China, India, Russia, Eastern Europe – are well represented.The last series of CAPTCHAs that we submitted to the services were intended to assess how fast workers could adapt to new CAPTCHA types. We exposed workers to a new type of CAPTCHA that is based on the Asirra CAPTCHA that was proposed in 2007. Asirra is a CAPTCHA based on identifying cats and dogs. Shown here is an example of the Asirra CAPTCHA (see right-hand image). Using the corpus of labeled cats and dogs provided on the Asirra website, we fashioned a CAPTCHA that we could submit to the services. Since their APIs only generally allow uploading one image at a time, what we did was we took that labeled corpus and we generated this CAPTCHA (see image on the left). We put the instructions in the 4 most prevalent languages – this is one image, so we just montaged a bunch of pictures together and then submitted to each of these services. So, our results: we got back about 15,000 responses from five of these services. ImageToText already had proven adaptable to new CAPTCHAs and solved this CAPTCHA with appreciable accuracy, close to 40%. BeatCaptchas also seemed to do exceedingly well (see right-hand image). To assess how well ImageToText workers learn, we plotted their error rate over two weeks that we submitted those images. Shown here (see right-hand image) is the decrease in error rate for the ImageToText workers: as you can see, the error rate drops steadily throughout the week, eventually reaching 40%. Thus, ImageToText at $20/1000 has a highly adaptable workforce. At approximately the same time that we deployed our custom version of the Asirra CAPTCHA, Decaptcher began offering an API specifically for Asirra. We were curious why, we didn’t think we influenced it, and we discovered that Microsoft had actually deployed Asirra on Club Bing on December 8, 2009. This is a screenshot from when they CAPTCHA’d you on that site (image on the left). Club Bing is a site where you go; you go there and play games, you earn their tickets, and then you can redeem those tickets for things like Xbox 360, Zunes, etc. So, you can see that there’s a good reason why you might want to abuse this service: you might want to get those products and resell them. Decaptcher implemented the API on January 17, 2010, meaning that roughly 5 weeks elapsed before the service responded to this new CAPTCHA type. Using the API, we observed the success rate of 46% among the responses that we got back. An additional 27% were off by one label. We observed that it takes roughly twice as long to solve these CAPTCHAs, which has factored into the retail price: I’m sorry I didn’t mention this, but they charge you $4 for every 1000 CAPTCHA solves. Decaptcher generally charges you $2 per 1000 CAPTCHA solves. And their workers do not earn more money.