CloudFlare’s CEO and co-founder Matthew Prince provides some additional details of the kerfuffle around Lulz Security’s activities during June-July 2011: the origins of their website traffic spikes and the different types of DDoS attacks CloudFlare was experiencing in that time frame. As a conclusion of this Defcon presentation, Sam Bowne talks on the defenses against denial-of-service attacks.I can talk about some things, I can’t talk about everything. I can talk about how these things affected us. I don’t want to get the host necessarily that they were using in trouble, so I’m not gonna be revealing their exact IP addresses. But let me tell you a little bit more about what happened over those 23 days.
This is the actual traffic to Lulz Security’s website (see stats on the image). Over those 23 days, they received a little over 18 million page views as people went to that site. You can see it peaked early, and then it trailed off since then.
What’s interesting is that we can actually look at what is just the attack traffic and break that down. And, you know, I would say that this attack traffic up until this spike (see image), kind of in the middle there was almost just background noise. It was not something that we were particularly concerned with. In fact, the three weeks that Lulz Security was on CloudFlare were actually three of the quietest weeks for denial-of-service attacks that we had seen, which is strange because a lot of people were saying that they were attacking them. There was this one spike in the center, but that seems to have been caused by a couple of very distinct events that they engaged in, and I’ll talk about what that is. And then I’ll talk about exactly what was the sort of attacks that we saw for Lulz, against Lulz, and what we did to defend ourselves; and then the ones that were sort of annoying to us.
So, one thing that was particularly interesting was on June 25th, this is the Jester. He publishes a post on his website (see screenshot). He spent a huge amount of time trying to figure out where the LulzSec site was hacked, and he proudly pronounced what has become gospel, which was that www.lulzsecurity.com was at 220.127.116.11, and lulzsecurity.com was at 18.104.22.168.
I know where the site was on June 25th, and I tell you it wasn’t there at all. In fact, they used seven different hosts over the course of 23 days. The original host was in Montreal, Canada. They were briefly in Malaysia, but it was in early June. It was that 111 address, that’s accurate. I don’t know where the other address comes from.
Most of the hosts that they used were actually U.S.-based hosts, including one large host that specifically specializes in DDoS mitigation. Ultimately they were using German hosting, that’s where they still are today.One thing that was interesting was that a lot of people claimed that they had found some way to knock Lulz Security offline, and they posted pictures online. This is actually a service that we offer at CloudFlare, which is if your back-end origin server goes down, then we actually show a cached version of this, and we put on an orange bar across the top that says: “You are viewing a cached version” – sort of like if you view as cache in Google (see screenshot).
What’s interesting is that while a lot of the world was claiming that they had done this, what I think actually must have happened is that the Lulz Security guys got kicked off their host, because for a brief period of time, for an about 36-hour period, what they did was they were actually pointing their IP address at 22.214.171.124, which is invalid, there is no host, there is no web server running there. I think they just picked a random IP address. And what that did was it caused our system to kick into the ‘always online’ mode. That actually caused that cached version to exist for a limited period of time until that cache expired. At that point, they pointed it back to a host for a short amount of time, then pointed back to a fake address, taking it up.
I am not aware of any person or any time when the Lulz Security site was actually knocked offline, in spite of the fact that a lot of people were trying to do that. On the other hand, they knocked a lot of people offline, which was interesting to watch.
With regard to a lot of the attacks that we saw, you know, we were really surprised. We had everyone on high alert. We were watching for big attacks to come in. And the attacks that we saw were generally significantly less than we would have expected.
Pissing off the hackers that populate Twitter is not nearly as dangerous as pissing off the Chinese cyber mafia or the Eastern European cyber mafia, or people that run really big extortion attacks – they run big DDoSes. These guys, you know, they are clever, but it’s not the same league.
We saw some Layer 7 attacks that were relatively harmless. Well, SlowLoris and some of those tools are interesting to attack an individual web server, and CloudFare was specifically designed not only to stop Layer 7 attacks dead, but we actually then record all the IP addresses that are committing those attacks. We actually are happy when people attack us over Layer 7.
The more annoying attacks for us were the Layer 3 and Layer 4 DDoS attacks. But we run a network which is an anycast network, and what that means is we have a bunch of machines, hundreds, and hundreds, and hundreds of machines, running in 14 different data centers all around the world, listening on the same IP address. So that tends to take DDoS attacks or high-volume attacks and spread them out over a very large surface area, which makes it much more difficult to launch something like that against us.
What is more interesting though about the annoying attacks that hurt us were a couple of different things. The first was that someone had a really big network and a lot of traffic, and they pointed almost all of it at us. And it happened that their network was geographically proximate to our San Jose data center, and so they were doing enough bandwidth to our San Jose data center. What we did was we took all of our clients other than Lulz Secutity and moved them to other data centers. No one ever noticed. But the San Jose data center for that period of time was only serving Lulz Security, kept them online though.
Another attack that was really interesting – it’s actually not a particularly big threat to most people, it was a threat to us – was using Google as a reflector. We have special rules that were in place for Google’s IP addresses in order to make sure that we’re never blocking legitimate crawler traffic from coming to us.
So, someone who is actually very clever found out that if they sent a lot of SYN1 requests with fake headers pointing back at our IP addresses to Google, Google would ACK2 back to those. And that actually created some issues for us internally. We found a pretty easy solution, we blocked the ACKs that didn’t have a SYN attached to them, and we called our friends at Google and said they would never get origin traffic coming from this, so they should just firewall it off, and that was solved within a few minutes. That was actually a clever attack that looked at the nature of how our system worked and challenged us based on that.
The last one which was the most annoying was when someone did a thorough scan of our IP address ranges and found some exposed router interfaces that were out there, and figured out the routers that we were using or just dictionary-attacked3 against the routers, we are not sure. And they were able to launch attacks that actually shut down some of our routers. They were able to bypass anycast because those were specific to that. The solution, again, was fairly straightforward: we just blocked those IP addresses off to the outside network. But it was the attack that actually caused us the biggest problem and knocked a couple of our routers offline for a couple of minutes.
But largely, again, when I compare the big attacks that we see when a client of ours gets a letter in the mail that says: “Hi, I’m a helpful Chinese government agency. By the way, we’ve detected on your network that someone is going to attack you. If you send us 10,000 dollars, you know, we can probably do something about it”. Obviously, that’s not a real Chinese agency, and they really can do something about it because they are launching it. Those are big attacks. These were relatively small by comparison.
So, a couple more things that were interesting. The first was when the Jester and all those guys were attacking. That’s sort of background noise pattern. What really started to trigger pissing people off was when the Lulz Security guys went after Minecraft4. And that was the real spike in traffic, and then the drop back off in traffic was caused when they stopped attacking Minecraft.
In fact, internally in our office, the biggest debates were in terms of whether we should drop them off our network or not came from the Minecraft aficionados who said: “You’re now causing me pain, and that’s not cool”. So, I guess the lesson is that if you are going to launch DDoSes against people indiscriminately, don’t pick on Minecraft.
I have very little information on who actually the Lulz Security folks are. I will say that one of the usernames that signed up for the CloudFlare account is very, very similar to one of the names that’s been arrested. I don’t know if that means that it’s just a coincidence or that they’ve actually been taken offline. We haven’t seen much activity move their host around; again, their website is down now. But it was an interesting 23 days watching all kinds of the attacks and all the world trying to take them down, seeing how we could help keep it up, for better or worse.
Sam Bowne: You know, I’ve been giving a lot of talks, and attack is easier than defense. So during my talks I used to say: “Oh, here is my new attack, it blows everything away, ha, ha, ha! And if you don’t like it – tough, you have to wait for Microsoft to patch it or something. Basically, you’re hosed”. This is a common message you’ll hear at Defcon or other conferences, but I am trying to move up. So, by the way, there are some defenses. I am trying to move into defense, which is tougher. Most of the time defense is difficult.
Now, if you wanna block those router advertisement floods, you can turn off IPv6 – that will protect you, but IPv6 is necessary and it does things that you probably want, like home groups and direct access. You can turn off Router Discovery with an NSH command at the command line. And that will mean that your machine does not listen, does not do anything when it gets RAs, and it will protect you against this attack. It will mean you have to put a static IPv6 address on it, which is probably the right thing to do on a server.
You can block it with the Windows Firewall, only except router advertisements from the authorized router, and that will protect your clients, although it is pretty easy to defeat that by just making rogue router advertisements that appear to come from that source address, but it will stop the attacks to some extent.And Cisco makes a switch with RA Guard. Cisco patched their own vulnerabilities with this. So if you buy a Cisco switch with RA Guard, you’re good. Anyway, you can evade that pretty easily by putting in fragmented router advertisements, which will go bypass Cisco RA Guard. So, for every defense there is another attack. But anyway, as far as defending, my conclusion have been for a long of time: the only reason your website is up is because nobody hates you. If even one person hated you, it would be down. That’s what the Jester proves. That’s why the Jester is so important to network security: he proves that just one angry man can take down a lot of websites – and you are helpless, basically. It’s not entirely true you are helpless, but the defense seems to be a little difficult to put in.
I tried playing with some defenses. You can use Mod Security. Now, in laboratory conditions, Mod Security’s latest version has an Anti-Layer 7 DoS feature, but all it does is stop too many connections from the same IP address, so it will save you from a test on your network but it won’t stop the Jester because he goes through Tor or some similar network, and all the attack packets come from different networks.
You can pay a service like Akamai to protect you, and they’ll use a few tricks to protect you.You can put in a Load Balancer (see screenshot). Load Balancer will protect your server by only letting complete requests make it to the server. But the Load Balancer itself will go down if you got enough traffic. It’s a defense, but it’s not a perfect defense. It took some four times as many packets to freeze the Load Balancer in my test, so it’s something.
You can also do things like counterattacks. If somebody tries to attack you with a botnet, you can point your DNS address back to their command and control server, and they’ll blow themselves away. That’s questionably legitimate, but it’s effective of course and it will work against flood attacks like Anonymous’ Low Orbit Ion Cannon.
But I was very pleased to observe CloudFlare here. Because I got the same talk to them about these horrible attacks, there is not much you can do. Now I am contacting people out of the blue that they have vulnerabilities exposed on Pastebin, to try to get them to fix that stuff.
They are typically small businesses that don’t know much, that don’t have any security team. I cannot tell them to purchase and implement an extra server to protect their server, but what I can do is tell to use CloudFlare which is a free service. That’s not too hard to do, and it really will protect you, and I was really pleased to observe that it really stopped the Jester. The Jester really wants to take them down, and he really can’t. This is the first time that I’ve seen us do that, and you can easily deploy it without having an expensive network security team.
1 – SYN request is a component of TCP connection establishment where the client sends a SYN to the server to further get a response.
2 – ACK is a signal passed between communicating processes or computers to signify acknowledgement, or receipt of response, as part of a communications protocol.
3 – Dictionary attack is a technique for defeating a cipher or authentication mechanism by trying to determine its decryption key or passphrase by searching likely possibilities.
4 – Minecraft is a sandbox-building independent video game written in Java originally by Swedish creator Markus “Notch” Persson. Minecraft allows players to build constructions out of textured cubes in a 3D world.