Generations of DoS attacks: some history and links to Jester, Anonymous and LulzSec

Sam Bowne from City College of San Francisco shares his expertise in the history of denial-of-service attacks, their technical aspects, and the major groups of hacktivists who use those for various purposes. This Defcon presentation starts with the classification of DoS attacks and the analysis thereof in terms of Jester, Anonymous and LulzSec attackers’ activity.

I am Sam Bowne, I teach at City College, San Francisco. I am here to talk to you about DoS attacks. I am gonna talk a little bit about the hacktivists who’ve used DoS attacks because I find them interesting, and they have dramatized how much damage you can do with the various kinds of DoS attacks, at the parallel of going to prison themselves for it, which is a drag, but anyway, it helps the rest of us sell security appliances, and it helps me entertain the students and keep them interested in knowing how these acts of defenses work.

So you’ll be participating as victims. Now, how many people brought a device to get killed? One, two, three – yeah, not very many, okay. That’s kind of what I thought, because Ryan who is setting up a wireless network says he probably cannot connect more than 40 or 50. I didn’t think there would be that many volunteers to get their device killed.

However, I was trying in the speaker room, and I believe this attack could be used to kill every machine at Defcon, from peer. I was gonna demonstrate a version of that – not so lethal – on the stage, but it wouldn’t connect it all in the prep room, so I decided to skip that for the moment. But if any of you are unscrupulous, you can try it.

I’ve got two guests with me: I’ve got Matthew Prince here who is gonna talk about his inside dealings with LulzSec¹, which I am very pleased to have. In fact, I met him because both of us were deplored as immoral evil people for helping LulzSec, because I retweeted some LulzSec tweets that pointed to stolen data which I thought was important. And he ran a service that they used to protect themselves from attacks. Another one of my guests – Ryan Carter – is gonna set up the network and ‘kill’ people who wish to volunteer to be DoS’ed with this attack, because we could learn some new vulnerabilities here. Now, they are not zero-days, because this – the attack I am using here – wasn’t written by me and it’s not new. It’s been known for a year. It’s just that an awful lot of people who manufacture devices don’t care and have not patched it. So, if anybody has exotic devices, it would be interesting to see if they are vulnerable.

Anyway, I’ll briefly outline what I want to show you. The DoS circus is about the history of this stuff and the attackers who have been using it. And then, I’ll talk about the three kinds of DoS: Layer 4 DDoS, where you use thousands of attackers to bring down one machine, usually distributed denial-of-service; Layer 7 DoS, where one attacker can bring down one server or more; and the Link-Local IPv6² router advertisement attack. I talked to you last year about IPv6, and I said it was gonna bring a lot of security problems – and so it has.

The DoS Circus

It has given us a time warp when a bunch of things designed in 1993 are now back on our networks, so the old tricks work again. This is not really an old trick, but it’s devastating and I’ll show it to you. You can kill all the Windows machines on a network from one attacker. And again, you only need a few packets per second to do it.

So, Julian Assange stirred everybody up by leaking U.S. secrets. He published this mysterious encrypted file as his insurance. And if he ever gets irritated enough at the fact that he is being held in a house arrest and perhaps gonna be deported and stuff, he can release the secret key and reveal something terrible, not yet specified.

So, this stirred up these Anonymous³ people that had gotten tired of just posting pictures of cats on 4chan, and decided to save the world through denial-of-service, which makes a lot of sense to them all, but not to me. So they started attacking. If there was anybody they could all agree to hate, they would blow them away. So they started with Scientology because it’s pretty easy to hate the scientologists.

Then it went on to other people – and eventually HBGary Federal⁴. This company’s CEO Aaron Barr was supposed to be here but he was issued a court order about 3 days ago, forcing him to not speak at the Panel and tell what really happened for the inside story here.

But anyway, in order to publicize his new government security contracting company, Aaron Barr said that he could find the people running LulzSec and expose them by doing a correlation of social networking. So what appeared in Twitter, he would correlate with what appeared in Facebook and elsewhere. And so, they decided to take him down, and it was extremely easy.

They got a team of Anonymous members. Now, Anonymous was a low-tech group, usually using really primitive tools. But a small number of them got together, who were relatively skilled compared to the others. And they decided to take these guys down. They found an SQL injection⁵ and took over the email server, and then they sent emails pretending to come from the owner of the company, asking him to please change the password, change the username and turn off the firewall. Thanks, that’s working now!

And once they were in, they took all the emails and dumped them on the Web, because the whole thing about these guys who later became LulzSec was complete irresponsibility. The fun thing is to take everything you were told not to do, and just do it, and then you laugh – ha, ha, ha!

So, what would happen if I just dumped your whole email log out, everything personal, hurting who knows how many innocent people that just had something to say about their medial conditions? So that would be a lot of fun, so that’s what they did. And they found a lot of real dirt in there. It looked like they were planning to do a lot of really nasty things from HBGary.

Then Anonymous decided to attack the Chamber of Commerce, having found out that they had a Drupal exploit, again showing more intelligence technically than the Anonymous had before, which had just used that Low Orbit Ion Cannon⁶, which is pretty primitive.

So the Jester (th3j35t3r) gets in here, using the demonstration of the power of a Layer 7 attack, although no one knows exactly what he does, he is truly secret, and I am guessing what he does. But from people who have been attacked and kept logs of his packets, they’ve told me that I am correct, that what he was doing is essentially using a SlowLoris⁷ attack with some variations.

His plan here is to be right-wing essentially, where Anonymous and LulzSec are left-wing. He is pro-military, he comes from the military, and he tries to punch back at anybody that he regards is endangering soldiers, like Julian Assange and Jihadist recruiting websites.

And he brings websites down with his tool, and then tweets about it. He is prominent on social networking, you can go chat with him, I’ve chatted with him. But he doesn’t have any partners, unlike LulzSec. He works alone, and therefore he hasn’t been caught yet. He understands military operational security. Nobody can betray him – something that LulzSec forgot.

Anyway, he brought down WikiLeaks single-handedly and held it down for more than a day. To prove it, I was chatting with him on IRC and he said: “Look, I can turn off the attack and let it come back up”. And it came back up. Then he said: “Now, I am taking it down again”. And it went down again. So that convinced me that he was really in control of the attack. Here is the Netcraft map of WikiLeaks going down for more than a day, thanks to the Jester (see image).

So that was his game, and then he decided to fight with Anonymous because Anonymous didn’t like him taking down WikiLeaks, and he has been focusing on them for about the last year, Anonymous and LulzSec blasting each other apart with the variety of tricks; he was putting on them denial-of-service.

And then, the Jester got mad at Westboro Baptist. Now, these guys are also pretty easy to hate. They have some ridiculous hatred of homosexuals, and they also picket funerals – basically, their profit method seems to be about being annoying until someone finally punches them in the face, and then sue. But the Jester decided to take them down, so he took down four websites with his tool, which he had ported to a cell phone. And from the single 3G cell phone, he says, he held down four websites for two months straight.

And I don’t doubt that because I know I can do it, and any of my students could do it, and you could do it if you just pay attention to this talk. It’s not hard. The SlowLoris attack runs on Windows, it’s not hard to do at all. And that’s how it goes.

Now, LulzSec continued on a rampage, hacking everybody in sight. At one point, they just opened up a telephone line and you could call them, and they would hack anybody you wanted. They hacked U.S. Government, Military, NATO, British Government sites. They dumped the contents of the Booz Allen Hamilton database. When the dumped out the Arizona cops is when I got really mad, because that was really important, they dumped out their names and password hashes and the logins for their emails.

And when they dumped out Booz Hamilton⁸ password hashes, that struck me as outrageous: 150,000 password hashes, half of them were cracked by the next day. So, all the top military, their names and passwords are now out there where anybody can use them, and they didn’t think much of that.

However, they also took down some game websites, which I didn’t even notice, but it seemed to be what really caused trouble for them.

They put up a website to announce all the stuff they took down and all the stolen data (see image). And then, they hacked PBS⁹ website and put up a silly thing there. I was pretty irritated by that, like – why would you hack PBS? Come on guys…

Anyway, now they’ve been caught, largely. Ryan Cleary was one guy kind of on the periphery of LulzSec, they caught him in June. Shortly after that, they caught T-Flow (see screenshot of news report excerpt) who was much more important to LulzSec. At the end of July 2011, they caught Topiary. So, they really are just British teenagers, and the attitude of taking down everything just for fun, you know, comes from just childish immaturity. You might wonder what makes them do this – they are just young and foolish, that’s why they think they can just take down every government website just for fun.

By the way, Jester and Sabu are supposed to be both here, they are both on Twitter claiming to be here, and they said they were at the pool yesterday. The Jester said he was here and Sabu said he was here. I kind of doubt it, but maybe they are, who knows.

Sabu is the main LulzSec person still at large. And why they assume to be on the way down is because his friends have are already been arrested, and this is what always happens: after they get the first one, they would find the rest, because they don’t have much of operational security.

Layer 4 DDoS (Many Attackers – One Target)

The technical part of this is you have a Layer 4 DDoS as the simplest kind of attack. This is what was used to take down MasterCard and Visa. They couldn’t take down Amazon this way. Anonymous tried this. This is a protest which involves many people.

So, the reason it does is because the tool they use is the Low Orbit Ion Cannon (see screenshot) which is just a network stress tester, and it doesn’t do much harm. So it takes a lot of people to bring down a website this way. But with the participation of 3,000 or perhaps 30,000 attackers, the number is not entirely clear, they were able to hold down MasterCard for more than a day, and many other sites.

And this is the kind of attack that Kaspersky was talking about when they interviewed him a while ago and asked him how many infected machines it would take to bring South Africa off the Internet completely. And he said it would take hundreds of thousands of infected machines to do that. And I know that’s false. I know it would take one 3G cell phone.

However, he is not thinking of that kind of attack, he is thinking of the Layer 4 attack, where it takes thousands of machines to take down one target. And it’s really nothing more than pressing F5 in your browser, F5, F5, F5… If enough people do that, the page goes down. It is a denial-of-service of a sort, it’s just a very weak primitive one.

Layer 7 DoS (One Attacker – One Target)

There are more powerful ones, like the SlowLoris attack that RSnake¹ came up with a couple of years ago. There were many previous versions of the same thing. Here you do something smarter.

Instead of sending a complete request to the web server, and just sending a lot of complete requests to the web server so it has to work too hard to serve them all up, you send it something that would jam up the web server, for instance an HTTP GET request to get a page from a server looking like this.

You have the Layer 2 information and Layer 3 information, and down here you got the GET, which is several lines of information. If you just send part of the GET and you never send the rest of it, then the network assumes that you are on some kind of unreliable network and the packets have been fragmented. And so I’ve got the first half of it, and the other half is still coming. So it waits for the other half and that ties up incoming lines. And it’s extremely powerful.

SlowLoris will freeze all available incoming lines, and all you need is about one packet per second to stop an Apache server dead.

R-U-Dead-Yet is another similar one, but it uses POSTs and affects IIS (see image). IIS is not affected by the SlowLoris attack with incomplete GET requests, but it is affected by incomplete POSTs requests.

There are other variations of it now. There is one using Keep-Alive DoS – that works, I tried that, it’s somewhat effective. It’s not as powerful as SlowLoris attack but it’s another way to send requests that make the server do a lot of work.

The Jester’s tool presumably uses one of these principles. It is called XerXeS (see screenshot). It is a graphical interface, looks like it runs on a bunch of Linux to me, but who knows. One important thing about Layer 7 attacks is you can run them through an anonymizer, so you don’t go to prison.

The Low Orbit Ion Cannon does not enjoy this feature because it has to send a lot of traffic from you to the other end. If you try to run it through the Tor³ network, it will just choke off your attack and bring down the Tor network, because it’s like flamethrower: it burns everything between you and the target. And with Layer 7 attack, it’s like a guided missile: it just sends a few packets that do not harm anything, and when it gets to the server – bang, the server becomes unavailable. So you can run it through an anonymizer, which is what he does, which means that not only can they not find out where it is coming from, but they also cannot protect from it by using a simple firewall rules that search by source address, because all the packets come from different source addresses. Although, if you block all Tor agent nodes, which you should all do, that will stop them from using Tor, and they would have to use something else like a botnet of compromised machines to do it, and that would make it a little harder.

But anyway, his tool starts, runs this thing through an anonymization network, and then brings down the target. And it independently does a series of tests to the target. When the target goes down, then it sends out twitts – ‘Tango Down’.

Link-Local DoS (IPv6: Router Advertisements)

Anyway, that’s where we were up to maybe 2 years ago, these things were running. The Link-Local DoS is much newer, with IPv6. You are using IPv6 if you have any version of any modern operating system, any modern version of Linux, any Windows Vista or Windows 7, or Windows XP – if turn on IPv6, although it’s not on by default. And you server as your domain controller, as your DNS server, as your email server are all using IPv6 whether you like it or not, unless you have gone out of your way to turn it off.

And like any other unwanted service, if you are not using it it’s opening you to the attacks. So, with IPv4, when a machine joins a network, unless you are weird enough to be using static IP addresses, which most people aren’t, your machine boots up and asks the router, a DHCP⁴ server: “I need an IP”, and it says “Okay, use this IP”. And then there is another back and forth to make sure nobody else is using that IP, and it’s the end of the game. There will be no further DHCP traffic until you restart that machine, or until a long time passes, like 4 days. That’s a PULL process: I need an IP, I ask for an IP.

But IPv6 is not normally done in that fashion. With IPv6, addresses are generally distributed by router advertisement. So the router pushes a router advertisement and says: “I am the router, everybody stop what you are doing and join my network now”. Everybody has to stop, make up an address and join the network. It’s a broadcast packet, although they say there is no broadcast in IPv6, but there is something called ‘multicast to all nodes’.

The difference between these things is still logical in nature, and I don’t intend to go into it. But the point is the router sends out one packet that goes to every node, and every node now has to join the network, which doesn’t seem that bad. Here is the router advertisement packet going to a multicast to a ‘multicast to all nodes’ address ff02::1 and telling people what network to join (see image). The problem is you can send out a lot of router advertisements. And when you do, the poor target joins all these networks. That would be alright, except that Windows is extremely inefficient in doing that.

Let me show you a few of these attacks. I should have some virtual machine set up. Well, this is how I do it in class with my students: I use virtual machines on an isolated network. Well, I’ll tell you a little more when I get there.

Let’s start with the old-fashioned attacks. Here I got a BackTrack 5 Linux machine, and it’s running as a web server. I put up a sample web page, so if I go to localhost and refresh, it is sending out that web page. If you run this page (localhost/server-status), you can see the status of your server. Okay, that’s the server status, and down here are the current connections (see screenshot). There is one connection waiting here and all the rest are available out of hundreds of connections available.

This server can handle hundreds of people viewing that web page. So, if I go to that test page and refresh, it should show up on server status page as another connection – and so it does. Now I have a couple of connections. So, now let’s attack this poor Linux machine from a Windows machine.

Low Orbit Ion Cannon attack in action

We will start with old-fashioned stuff – Low Orbit Ion Cannon. And Low Orbit Ion Cannon is here, the thing that Anonymous people use as a short way of going to prison. So I put the IP address and press ‘Lock on’ button, and IP address appears as a selected target, and I can now do different kinds of attacks here. In addition to sending you to prison, Low Orbit Ion Cannon isn’t very well written, doesn’t let you see what you need to see well. So, I am going to send an HTTP request and I am charging my laser, and now it’s sending stuff, sending complete requests back to my poor target.

So, my poor virtual machine has now started showing new connections on server status page. It’s filling up with a bunch of C’s. Now, those C’s are connections at the web server. It is gradually filling up here (see screenshot), so it is using up all that the web server can do. And again, what it’s doing is complete connections: they form a connection, they download that little web page, and then they wait to time-out. This does fill up all the connections and makes the web server unavailable, but it does it in a very weak way because each connection terminates normally and then just ends its time normally, so it only ties it up for a couple of seconds. That’s what this one does.

SlowLoris attack process

Let’s now get to SlowLoris which is much more powerful. I have to put that IP address in the ‘HTTP attack (slow headers and slow POST)’ interface (see screenshot) and push ‘Run attack’. Then I go back to server status page – and see the connections are filling up with R’s. Those are pending requests; each one of those will take 400 seconds to time-out by default. So, you don’t need to send very many of them, and it uses up all available incoming lines – and this server is toast. That’s the Slowloris attack, and the HTTP POST¹ attack is similar. It’s very powerful and very dangerous.

Link-Local IPv6 attack implementation

Anyway, now that I have shown how to kill Linux with Windows, let’s go the other way, with a more powerful attack. Let me set up my poor Windows machine to show you the evil that is about to happen to it. So, if I go into ipconfig² you see this machine is an ordinary Windows machine. I put on a static address 2::2 in IPv6, and it’s got IPv4 address, and really not much else going on. Using the Task Manager is a good way to see the damage that’s gonna happen to this machine. The CPU usage is now at 0%.

Now, if I send some IPv6 packets here, I am gonna do fake router6 first, let’s send it to def:c0::. And now it is sending some packets advertising that network; all the devices on that network have been commanded to join it. And there it is – it’s made the address start with def:c0:: (see screenshot). Now, this is what’s supposed to happen when you add a router in a normal course of events: I add a router, it advertises its prefix, everybody joins – and the game is over.

But if I send a flood of unwanted packets at the rate of 100 per second, we see that CPU usage on our target machine is 100%, and it’s just gonna sit there at 100% for a long, long time. And what’s worse is it kills so bad that you can’t see the address.

If you go into ipconfig and stop it really fast, this will actually respond without waiting forever, and you can see what it’s done: it’s joined all these networks – page after page of networks (see screenshot). That’s what it’s doing. And it’s still adding more to that list, at the rate of about 5 per second.

So, this is alright, but when I first tried it, I ran it for a while and nothing seemed to happen, and all of a sudden – hey, my Windows machine doesn’t respond at all. What happened? Well, this is no fun, my students don’t learn anything if they can’t look at the damage. So I thought this was a bad project, what do I do? And then I thought – hey, wait a minute, this would kill the domain controller, and the email server, and everything. This is really bad. This is so bad I cannot tell my students at all. I’d better tell Microsoft quietly.

So I sent out a Tweet and said: “Hey, this attack hurts Windows 7”. And then I said: “I need a security contact inside Windows”. I added some other people on Twitter, which immediately gave me good people inside Microsoft, and they sent me to the right people. Within two days, I had an official answer from Microsoft, saying: “Yes, Van Heusen told us about that a year ago, and we don’t care. We’re not gonna do anything about it for current versions of Windows. We do not care that Windows Vista, Windows 7, Windows Server 2003, Windows Server 2008, XP are all gonna die at the drop of a hat. We may put it in Windows 8 or Windows 9 or something, if we have nothing better to do”. I said: “Fine! If you’re gonna be that way, I’ll tell the whole world about it”.

Judging from my research, Windows machines are vulnerable to this, and one version of BSD Unix is vulnerable to this attack. You can run the attack on Mac too, the Mac is the host. It joins some of these useless networks, but it doesn’t join them all – about first 10 and no more. It has the sense to ignore all router advertisements after the first 10, for some period of time, which is a pretty good defense. And that’s what I think Microsoft should do in Windows, but they are not interested in my opinion.

I gave it to my students for homework and said: “Use an isolated network, don’t kill every machine at the college – because you could kill every machine at the college, including our servers and everything else”. And my students didn’t kill the whole college with it, which is very nice of them. Therefore I am still working there, I am not on the street with a tin cup.

Anyway, that’s the power of that attack, and we should as well talk a little bit about defenses. But I think before I do that, I am gonna hand it over to Matthew here.

LulzSec: Behind the Scenes

Mathew Prince: So my name is Mathew Prince, and I know Sam, we both live in San Francisco. Sam is the only person I know who can make running DDoS attacks seem charming. We both got sort of dragged into Lulz Security kerfuffle reluctantly. I am gonna tell you the story of how I got dragged into it and talk to you about some of the DDoS attacks that we saw during the 23 days that they were active, and then what we did to stop them.

So, on June 2, 2011, at about 16:54 GMT the Lulz Security Twitter account announced that they had finally gotten around actually making a web page. What was pretty amazing was that within about 15 minutes that web page was knocked offline by a fairly significant denial-of-service attack. I don’t know the details of this particular attack because we hadn’t been involved yet.

About an hour after the web page was first announced, LulzSec announced on their Twitter account that they had actually solved this problem. The only thing that had changed, as far as I’ve been told, is that 9 minutes earlier they signed up for CloudFlare³ – a service that makes websites faster, we protect them from some attacks, but don’t really think of ourselves as an anti-DDoS service. So it was somewhat of a surprise for the Lulz Security people to do that.

It was even more of a surprise when an hour later Lulz Security sent out a message to me, saying: “We love your service so much, can we exchange rum for a free Pro account?” I had no idea who Lulz Security was at this point, so I tweeted back a tweet which my legal council has told me to remove, which said: “It depends on how many cases and how good the rum is”. They never sent the rum, and we never gave them a Pro account, but CloudFlare is free and thousands of sites sign up for it every single day, and we typically don’t have problems with them. We had some more issues with these guys. So, over the course of the next 23 days, they wreaked mayhem in lots of different ways, and finally on June 25 they called it quits.

What was interesting is that the way CloudFare works is for a reverse proxy, so all of the traffic which goes to Lulz Security passes through our network first, which has two significant effects. The first is – anyone who attacked Lulz Security was attacking us, so that was amusing. And then secondly, it meant that Lulz Security was actually able to hide where their origin was, where they were actually hosting from. That’s a side affect of how our system is designed, but it was one that they used to create affect.

Sam actually contacted me a little while ago, and said he was going to do a talk on DDoS and asked me whether I would be willing to share some information about it. And again, we have legal council and we are a real company, and we have a privacy policy. Even if you are an internationally wanted cyber criminal, we try to respect the privacy policy, so I wrote the following email to the email account that we had on file for Lulz Security, on July 2, right after they called it quits: “Hey, I’ve been invited to talk about this at Defcon, would you mind?” And I didn’t hear anything for quite some time. And then 11 days later, someone by the name of Jack Sparrow wrote: “You have my permission”.

I can talk about some things, I can’t talk about everything. I can talk about how these things affected us. I don’t want to get the host necessarily that they were using in trouble, so I’m not gonna be revealing their exact IP addresses. But let me tell you a little bit more about what happened over those 23 days.

This is the actual traffic to Lulz Security’s website (see stats on the image). Over those 23 days, they received a little over 18 million page views as people went to that site. You can see it peaked early, and then it trailed off since then.

What’s interesting is that we can actually look at what is just the attack traffic and break that down. And, you know, I would say that this attack traffic up until this spike (see image), kind of in the middle there was almost just background noise. It was not something that we were particularly concerned with. In fact, the three weeks that Lulz Security was on CloudFlare were actually three of the quietest weeks for denial-of-service attacks that we had seen, which is strange because a lot of people were saying that they were attacking them. There was this one spike in the center, but that seems to have been caused by a couple of very distinct events that they engaged in, and I’ll talk about what that is. And then I’ll talk about exactly what was the sort of attacks that we saw for Lulz, against Lulz, and what we did to defend ourselves; and then the ones that were sort of annoying to us.

So, one thing that was particularly interesting was on June 25th, this is the Jester. He publishes a post on his website (see screenshot). He spent a huge amount of time trying to figure out where the LulzSec site was hacked, and he proudly pronounced what has become gospel, which was that www.lulzsecurity.com was at 204.197.240.133, and lulzsecurity.com was at 111.90.139.55.

I know where the site was on June 25th, and I tell you it wasn’t there at all. In fact, they used seven different hosts over the course of 23 days. The original host was in Montreal, Canada. They were briefly in Malaysia, but it was in early June. It was that 111 address, that’s accurate. I don’t know where the other address comes from.

Most of the hosts that they used were actually U.S.-based hosts, including one large host that specifically specializes in DDoS mitigation. Ultimately they were using German hosting, that’s where they still are today.

One thing that was interesting was that a lot of people claimed that they had found some way to knock Lulz Security offline, and they posted pictures online. This is actually a service that we offer at CloudFlare, which is if your back-end origin server goes down, then we actually show a cached version of this, and we put on an orange bar across the top that says: “You are viewing a cached version” – sort of like if you view as cache in Google (see screenshot).

What’s interesting is that while a lot of the world was claiming that they had done this, what I think actually must have happened is that the Lulz Security guys got kicked off their host, because for a brief period of time, for an about 36-hour period, what they did was they were actually pointing their IP address at 2.2.2.2, which is invalid, there is no host, there is no web server running there. I think they just picked a random IP address. And what that did was it caused our system to kick into the ‘always online’ mode. That actually caused that cached version to exist for a limited period of time until that cache expired. At that point, they pointed it back to a host for a short amount of time, then pointed back to a fake address, taking it up.

I am not aware of any person or any time when the Lulz Security site was actually knocked offline, in spite of the fact that a lot of people were trying to do that. On the other hand, they knocked a lot of people offline, which was interesting to watch.

With regard to a lot of the attacks that we saw, you know, we were really surprised. We had everyone on high alert. We were watching for big attacks to come in. And the attacks that we saw were generally significantly less than we would have expected.

Pissing off the hackers that populate Twitter is not nearly as dangerous as pissing off the Chinese cyber mafia or the Eastern European cyber mafia, or people that run really big extortion attacks – they run big DDoSes. These guys, you know, they are clever, but it’s not the same league.

We saw some Layer 7 attacks that were relatively harmless. Well, SlowLoris and some of those tools are interesting to attack an individual web server, and CloudFare was specifically designed not only to stop Layer 7 attacks dead, but we actually then record all the IP addresses that are committing those attacks. We actually are happy when people attack us over Layer 7.

The more annoying attacks for us were the Layer 3 and Layer 4 DDoS attacks. But we run a network which is an anycast network, and what that means is we have a bunch of machines, hundreds, and hundreds, and hundreds of machines, running in 14 different data centers all around the world, listening on the same IP address. So that tends to take DDoS attacks or high-volume attacks and spread them out over a very large surface area, which makes it much more difficult to launch something like that against us.

What is more interesting though about the annoying attacks that hurt us were a couple of different things. The first was that someone had a really big network and a lot of traffic, and they pointed almost all of it at us. And it happened that their network was geographically proximate to our San Jose data center, and so they were doing enough bandwidth to our San Jose data center. What we did was we took all of our clients other than Lulz Secutity and moved them to other data centers. No one ever noticed. But the San Jose data center for that period of time was only serving Lulz Security, kept them online though.

Another attack that was really interesting – it’s actually not a particularly big threat to most people, it was a threat to us – was using Google as a reflector. We have special rules that were in place for Google’s IP addresses in order to make sure that we’re never blocking legitimate crawler traffic from coming to us.

So, someone who is actually very clever found out that if they sent a lot of SYN¹ requests with fake headers pointing back at our IP addresses to Google, Google would ACK² back to those. And that actually created some issues for us internally. We found a pretty easy solution, we blocked the ACKs that didn’t have a SYN attached to them, and we called our friends at Google and said they would never get origin traffic coming from this, so they should just firewall it off, and that was solved within a few minutes. That was actually a clever attack that looked at the nature of how our system worked and challenged us based on that.

The last one which was the most annoying was when someone did a thorough scan of our IP address ranges and found some exposed router interfaces that were out there, and figured out the routers that we were using or just dictionary-attacked³ against the routers, we are not sure. And they were able to launch attacks that actually shut down some of our routers. They were able to bypass anycast because those were specific to that. The solution, again, was fairly straightforward: we just blocked those IP addresses off to the outside network. But it was the attack that actually caused us the biggest problem and knocked a couple of our routers offline for a couple of minutes.

But largely, again, when I compare the big attacks that we see when a client of ours gets a letter in the mail that says: “Hi, I’m a helpful Chinese government agency. By the way, we’ve detected on your network that someone is going to attack you. If you send us 10,000 dollars, you know, we can probably do something about it”. Obviously, that’s not a real Chinese agency, and they really can do something about it because they are launching it. Those are big attacks. These were relatively small by comparison.

So, a couple more things that were interesting. The first was when the Jester and all those guys were attacking. That’s sort of background noise pattern. What really started to trigger pissing people off was when the Lulz Security guys went after Minecraft⁴. And that was the real spike in traffic, and then the drop back off in traffic was caused when they stopped attacking Minecraft.

In fact, internally in our office, the biggest debates were in terms of whether we should drop them off our network or not came from the Minecraft aficionados who said: “You’re now causing me pain, and that’s not cool”. So, I guess the lesson is that if you are going to launch DDoSes against people indiscriminately, don’t pick on Minecraft.

I have very little information on who actually the Lulz Security folks are. I will say that one of the usernames that signed up for the CloudFlare account is very, very similar to one of the names that’s been arrested. I don’t know if that means that it’s just a coincidence or that they’ve actually been taken offline. We haven’t seen much activity move their host around; again, their website is down now. But it was an interesting 23 days watching all kinds of the attacks and all the world trying to take them down, seeing how we could help keep it up, for better or worse.

Defenses against DoS attacks

Sam Bowne: You know, I’ve been giving a lot of talks, and attack is easier than defense. So during my talks I used to say: “Oh, here is my new attack, it blows everything away, ha, ha, ha! And if you don’t like it – tough, you have to wait for Microsoft to patch it or something. Basically, you’re hosed”. This is a common message you’ll hear at Defcon or other conferences, but I am trying to move up. So, by the way, there are some defenses. I am trying to move into defense, which is tougher. Most of the time defense is difficult.

Now, if you wanna block those router advertisement floods, you can turn off IPv6 – that will protect you, but IPv6 is necessary and it does things that you probably want, like home groups and direct access. You can turn off Router Discovery with an NSH command at the command line. And that will mean that your machine does not listen, does not do anything when it gets RAs, and it will protect you against this attack. It will mean you have to put a static IPv6 address on it, which is probably the right thing to do on a server.

You can block it with the Windows Firewall, only except router advertisements from the authorized router, and that will protect your clients, although it is pretty easy to defeat that by just making rogue router advertisements that appear to come from that source address, but it will stop the attacks to some extent.

And Cisco makes a switch with RA Guard. Cisco patched their own vulnerabilities with this. So if you buy a Cisco switch with RA Guard, you’re good. Anyway, you can evade that pretty easily by putting in fragmented router advertisements, which will go bypass Cisco RA Guard. So, for every defense there is another attack.

But anyway, as far as defending, my conclusion have been for a long of time: the only reason your website is up is because nobody hates you. If even one person hated you, it would be down. That’s what the Jester proves. That’s why the Jester is so important to network security: he proves that just one angry man can take down a lot of websites – and you are helpless, basically. It’s not entirely true you are helpless, but the defense seems to be a little difficult to put in.

I tried playing with some defenses. You can use Mod Security. Now, in laboratory conditions, Mod Security’s latest version has an Anti-Layer 7 DoS feature, but all it does is stop too many connections from the same IP address, so it will save you from a test on your network but it won’t stop the Jester because he goes through Tor or some similar network, and all the attack packets come from different networks.

You can pay a service like Akamai to protect you, and they’ll use a few tricks to protect you.

You can put in a Load Balancer (see screenshot). Load Balancer will protect your server by only letting complete requests make it to the server. But the Load Balancer itself will go down if you got enough traffic. It’s a defense, but it’s not a perfect defense. It took some four times as many packets to freeze the Load Balancer in my test, so it’s something.

You can also do things like counterattacks. If somebody tries to attack you with a botnet, you can point your DNS address back to their command and control server, and they’ll blow themselves away. That’s questionably legitimate, but it’s effective of course and it will work against flood attacks like Anonymous’ Low Orbit Ion Cannon.

But I was very pleased to observe CloudFlare here. Because I got the same talk to them about these horrible attacks, there is not much you can do. Now I am contacting people out of the blue that they have vulnerabilities exposed on Pastebin, to try to get them to fix that stuff.

They are typically small businesses that don’t know much, that don’t have any security team. I cannot tell them to purchase and implement an extra server to protect their server, but what I can do is tell to use CloudFlare which is a free service. That’s not too hard to do, and it really will protect you, and I was really pleased to observe that it really stopped the Jester. The Jester really wants to take them down, and he really can’t. This is the first time that I’ve seen us do that, and you can easily deploy it without having an expensive network security team.