Life Inside a Skinner Box 6: Implementing an Automated System the Right Way

The scientists are highlighting ways to implement automated law enforcement without having to suffer the consequences of malfunction and possible abuse.

Attempting to defeat the device Lisa Shay: So, what can we do about this? Obviously there are countermeasures that are available for all different kinds of problems. Greg and I gave a talk at the HOPE 9 conference earlier this month about a taxonomy of countermeasure approaches.

Possible countermeasures for surveillance And in this community we love to defeat the device. We are all about reverse engineering the firmware and repurposing devices for our own needs, and that’s great, and that’s absolutely a way of providing countermeasures, or man-in-the-middle attacks on the network, or Defeat the Processing. How securely is that database recording all our data? Can we temper with it and make it look different for us? Those are fun and those are exciting engineering challenges, but even more important, we assert, is the countermeasures, or the influence on the actors, the decision makers, the people who decide to build these systems, to emplace these systems, and potentially to regulate these systems, because if we can prevent a bad system from being emplaced in the first place, an ounce of prevention is worth a pound of cure.

And that takes us out of our comfort zone, because those are dealing with real people, not with inanimate objects, but that’s a vital task to engage the media, to engage policy makers, to engage law enforcement officials, to engage the people who design, build and test these systems.

Greg Conti: Because once the system’s in place, the local economy, the local leaders become addicted – particularly if it’s profitable – to the financial resources that it brings in, and getting it to slodge is going to be far more difficult.

Are sensors registering your activities properly? Lisa Shay: Yeah, better to prevent it than to try and remove it after the fact. And then also you have to worry about competing sensors: when you have these different sensor mechanisms, how are they calibrated? How often are they maintained? How regularly are they maintained? Because if you have different sensors that detect different things about your activities, which one is right?

Cruise control speed graph And then we have to look at, really, how these laws are written and how they would be algorithmically implemented. This is a graph of data that I took from my 2006 Prius showing vehicle speed over a period of about 5 minutes (see left-hand image). In this test I set my cruise control to 42 mph, which is the pink line going straight across the graph. At the very beginning I was going a little bit downhill, and you can see the speed rise. And then I went a little bit uphill, and the speed dropped below 42. And then I was on some relatively flat terrain, and yet the speed is still bouncing back and forth. Why is that?

Well, speed is inherently an analog quantity, it has an infinite variability. But the computers on board our cars are digital systems, so they’re doing analog-to-digital conversion, and inherently there’s some quantization error involved. And it turns out that the computer system on board my Prius has a quantization window of about 0.6 mph. So it turns out that that computer will never read exactly 42 mph; it’s going to read 41.6, 42.2, 42.8 and so forth, even though the number that it actually spits out is 4 digital behind the decimal place, so it will be like 41.6374 mph. You think it’s really accurate, but it’s not.

Inherently there’s some quantization error involved.

And so, if you just look at this little graph, if the speed limit was exactly what that red line was, there’s about 17 times within 3 minutes that I violated the speed limit, even though my cruise control was set at the speed limit. So, would I get 17 speeding tickets? I hope not, but the law has to take that into account: if anyone of us who is tasked with writing code to enforce the speed limit law, how would you do it? Would you have this kind of level crossing scheme where every time you cross the speed limit on the upward trend, you counted that as a violation? Or would you have some kind of sliding window scheme that said: “Only if the level was crossed for 500 milliseconds or 300 milliseconds,” would you count that as a violation? If there’s three violations within a certain period of time, does that count? Is that 3 or is that 1? So there’s lots of devils in the details.

Woody Hartzog: And I should add that there’s no current infrastructure emplaced in the law to respond to that, because of course these laws were not written with algorithmic precision in mind. For example, take trespassing: it’s a violation of the law to trespass, but if you were tasked with making sure that if a GPS device could probably tell whether you’re on someone else’s property or not, how long do you have to be on the property before it’s a trespass?

Is it a few seconds? What if you’re walking down the boundary and one foot touches over it? How far deep into the property do you have to be to be trespassing? There are all of these little decisions that we make as judgment calls all the time using discretion and deciding whether to enforce the law, that then have to be encoded, and if you make an error, then all of a sudden you’ve systematized the error of the law.

Lisa Shay: That opens up a wide range of research topics. This is an unsolved problem and we’re trying to prevent problems, so the community really has to engage in critical analysis of what are the metrics to decide risk vs. benefit. At what point is it worth implementing an automated system? How much benefit do you have to derive vs. what kind of cost?

These systems need to be designed for transparency and accountability.

And then these systems need to be designed for transparency, they need to be designed for accountability. We submit that they should have manual overrides in them: if the car is going to prevent me from violating the speed limit, in theory that sounds like a great thing, but what if I am running to the emergency room? I’d like to be able to get there quickly. How many of you saw the video footage from the Japanese earthquake, when there was this huge tsunami wave, and there was this little car riding down the road trying to outrun the tsunami wave. If my car couldn’t go past, like, 30 mph on that road, that wouldn’t be good.

So we’re going to have manual overrides, and we want to build in the security systems. You’re all going to find the flaws, and hopefully you’ll tell us, and hopefully we’ll be able to do something about it. But we want to be able to build in some minimal level of security.

Automation issues are sort of similar to environmental problems in their effects And the thing is, this isn’t going to happen overnight. This sort of problem is similar to environmental problems. A little bit of pollution here and there, and then suddenly you wake up one morning and your river’s on fire. You know, it’s the same kind of thing here: you get a little sensor here, a speed camera there, a new computer system in the police department, and then the next thing you know: we’re living in a police state.

Be careful what you build So, be careful what you build, and, in summary, these systems can be implemented; there’s sensor technology out there right now that has the potential to automate a lot of the law enforcement process. And if it’s not done well, we could have some really serious unintended consequences, and you all in the audience are in a unique position to help avert these catastrophes.

References If you’re interested, PDF on these slides has all the references (see right-hand image). We’ve done a talk at HOPE on countermeasures, and we’ve also written a paper for the We Robot Conference that talks about some of this in more detail. And we’d really like to thank John Nelson and Dom Larkin, who were two colleagues that collaborated with us on the We Robot project, who were unable to be here with us today. Thank you very much!

Read previous: Life Inside a Skinner Box 5: The Mixed Blessing of Perfect Law Enforcement

Like This Article? Let Others Know!
Related Articles:

Leave a comment:

Your email address will not be published. Required fields are marked *

+ 9 = 10

Comment via Facebook: