Having presented the main material of the presentation, Ben Hagen gets to take some questions from 29c3 attendees regarding security issues at Obama campaign.
Host: Ok, thank you! If anyone has a question, there are three mikes in the room. So, please stand up if you have a question.
Question: Hi. First of all I’d like to say that a few weeks back I read an article comparing your work with the work of the opponent about keeping the system running and preventing collapse, and you guys did a great job, kudos for that. My question is: when you work with security, one of the things you always encounter is the lack or not doing well of security assessment or threat assessment. Who is in charge of deciding what are the likely threats or what are the possible threats, and who decides which ones of them should be addressed?
Ben Hagen: I think it was really advantageous for us to have the developer community that we had; I think they had a very realistic view of what kind of threats were being faced on the Internet, and they could kind of integrate that into their own development lifecycle. In terms of the organization as a whole, I would say that our Chief Technology Officer and campaign management were kind of in charge of assigning risks to the larger threats that we would face, so generally, if I found an issue, I’d bubble it up to that level, and they would make decisions based on what kind of realistic threat approached the organization, or the impact that it could have in the near term.
Question: Were you compromised a lot? Like they said: “We understand, but it’s not going to happen.”
Ben Hagen: We warned people a lot. We weren’t aware of any actual compromises that we had of our data or anything like that, which is very fortunate, I think, in that regard. But I think we were constantly receiving threats from people that were very realistic, and I think the communication up the channel in terms of that was very important.
Question: When was the internal social network most effective during the campaign, maybe in some cities or some events? And what were the main issues in terms of security?
Ben Hagen: So, talking about the social network, we had an application called Dashboard; that was our internal tool for social networking, it was its own social network with its own accounts, its own login system, that kind of thing. It was very effective at organizing people at the micro level. So, people would join up onto the social network, and they would find groups that they had affinity for, for example, by interests, by location, by neighborhood, etc. It was very effective at cementing those relationships.
In terms of the social network, I think, the threats we faced were mostly fraud-related or messaging-related. We had big issues with people making fake accounts and spamming the entire board and sending messages to lots of people. So, moderation became a very important part of what we did, not with the goal of censorship, but with the goal of keeping the riffraff out of there when it tried to cause some sort of issue with it. So, I think that kind of fraud was kind of the bigger issue.
Host: We have a question from the Internet. The question is: how much did Obama himself make the job more difficult or easy? And did he request any special features?
Ben Hagen: Obama played more of an advisory role throughout the entire election. He was, obviously, the sitting President. I think most of the decisions were made by the campaign management, which is the campaign manager and senior advisors. He did play a role in terms of the messaging that we put out, general policy and that kind of thing, but he was hands off in terms of technology. He and the campaign management let us build what we thought would answer the problems that they were having. So, I think it was great to have him as a figurehead, but in terms of day-to-day business he didn’t play a huge role.
Question: First of all, thanks for your talk about the situation where you had to close down a network a little more. We get a lot of talks here about open networks, and everyone can connect with everything. I like the point that you presented here. One of the things you mentioned was that you did some internal training, like tricking people into clicking on attachments. I would think that you could get an angry mob against you quite soon. Can you tell us about that?
Ben Hagen: I think it’s important to make it more like a game, as opposed to something that creates a vindictive hatred of you or something like that. We played coy about it a lot, so we’d send these kind of things out. The training was that if you received an email that you were skeptical of, you should contact the help desk. You shouldn’t click on anything, you shouldn’t send it to anybody; you should contact the help desk immediately and get the problem resolved. I think you’re right, I think it’s a dangerous game to play with people. We kind of hid behind the fact that nobody was sure what was and wasn’t training for us. So, people knew we were actually getting these threats, we would send out information if we had a particularly wide campaign committed against us, but in terms of what we did internally, we never really let people know that there was a wide-scale thing that we were doing.
Question: At one point you mentioned that during the campaign you used AWS extensively, which is understandable by the dynamic of the network and the infrastructure. You said something about 2000 nodes in the network. Were these servers or client? What was the ratio between them?
Ben Hagen: Those are a number of servers. For example, on Election Day we had over 2000 servers grown into AWS across multiple zones, all serving the applications that we were doing. In terms of the ratio, I’m not exactly sure what that was. We had pretty aggressive scaling limits set on stuff, so things would scale up pretty readily. On Election Day we kind of threw caution to the wind and money to the wind and just said: “Scale everything up, get as much as we can,” so the ratio was probably still thousands to a machine, but we had a ton of traffic going through.
Question: And how many clients were there?
Ben Hagen: We had a lot. I think the best statement regarding that is probably what our DevOps person said: 8.5 billion requests; on a given day we had several million requests to our main websites.
Host: One more question from the Internet: how many people were volunteers and how many people were full-time employees? And did that present an additional challenge?
Ben Hagen: Sure, I think it did present a challenge just in terms of the disparity, in terms of experience, and how much time they spent on the campaign. So, obviously, if you spend a lot of time there and you are actual staff, I think you’re motivated in different ways than a volunteer. They’re both great and incredibly helpful, but there’s certainly a challenge in rectifying that difference.
In terms of full-time staff at the headquarters, I believe we had between 500 to 700 at different parts of the campaign; that’s out of the one location in Chicago. I think across the country the number was more in the low thousands or something like 2000 or 3000 paid staff in the country. In terms of volunteers, that’s a lot more; I think a lot of it depends on what you call a volunteer. I know the bigger end of numbers; people say we had 2.2 million volunteers if you take into account people with online accounts, or people that had shared information. Realistically, it’s probably more in the tens to hundreds of thousands.
Read previous: Securing the Campaign 5: Application Assessment