Ethical hacker and security researcher with Trustwave Nicholas Percoco keynotes at RSA Conference 2013, addressing nuances of the present-day cybercrime.
Nicholas Percoco: Thank you very much! Good afternoon! We’ll be spending the next 30 minutes or so talking about the lifecycle of cybercrime.
We live in a world which has over 7 billion people. Now, if you put on the hat of a financially motivated cybercriminal and look at this population, not everybody here has information that you’d want to steal. When narrowed on that scope just a little bit, you’d run into about a billion people. These people have bank accounts, they have credit cards, they have social media accounts, they have data they store in databases online – and these are people that you may want to go after if you are a financially motivated cybercriminal trying to make money, trying to make a living.
Unfortunately for the criminals, if you have to go one by one and pick off each of those individuals, it takes a long period of time. Some criminals go about doing that, but some of the large criminal gangs that are going after information – they don’t go about it that way. They instead turn and target businesses, or organizations that store, process, transmit, maintain databases that contain that information; and there’re about 25 million of those business in the world.
Another way to look at these businesses: for every one business that has databases, that has information, that has data that criminals may want to go after, there’re about 40 people that contribute to adding and put that information into those databases. And if you think about how criminals look at this situation and how they feel about going to market, running their businesses and trying to infiltrate and exfiltrate data out of organizations, there’re different ways to look at this. But I’m going to actually show you, I’m going to tell you a story about how I felt when I was about 9 years old. I’m going to draw some comparisons here.
When I was 9 years old I went on a fishing trip, went off into the middle of the Gulf of Mexico, had my father and my grandfather by my side, and we were going for a Saturday fishing trip. And so, when we were going out in the boat, the boat’s going across the water, and it explained to me that we’re going to go fishing today. But you may not catch any fish. Being 9 years old, I thought this was really confusing: we’re going fishing, I have a fishing pole, we have bait – why might I not catch any fish? My father’s telling me this…But what was explained to me by my grandfather is that when you go fishing, sometimes you don’t catch anything.
So the boat stopped, they dropped anchor, and they told us we could start fishing. I’m putting a piece of squid on the hook and dropping it into the water – and within a minute I caught a fish. I looked at my father and my grandfather, saying: “Well, what’s going on? You just told me I wasn’t going to catch any fish today, and I just caught one!” And then my grandfather put bait on his hook, dropped it into the water, and within a couple of minutes he caught a fish. I was like: “Wow, this is great!”
And then we were glancing on the boat, and there were other people on the ship reeling in fish, left and right. And at the end of that day of fishing, we caught about 70 fish between the three of us. So this was sort of a really strange situation for me: I was being prepped, basically, to fail here, and we end up catching 70 fish!
And so, if you are a cybercriminal, a financially motivated cybercriminal, and you are looking to target victim organizations, they are not hard to find. These victim organizations are not, you know, scattered out through the sea; you don’t have to really try to find them – they are everywhere, and we know this.
We performed 450 cybercrime investigations, forensic investigations last year for businesses all over the world. And these businesses were not just in the United States; they weren’t just located in the UK or Australia. They were all over the planet, and they had many things in common. One thing they had in common, or a portion of them, is that many of them lived below of what Wendy Nather of the 451 Group had coined the “security poverty line“.
These are organizations that have very little security controls in place, or, really, an inability to defend against various attacks. Now, many of these organizations often don’t know when they’ve actually been breached. We know from our studies last year and from our investigations that the average time for an organization to actually identify a breach and start stopping the problem is about 210 days. And then even beyond that, there are some organizations, the ones that are doing a good job – it takes them about 10 days. For about 5% it takes about 10 days to identify a breach.
But in the worst case scenarios, we saw about 20% of organizations took greater than 2 years to identify when they’ve been infiltrated. And actually a small percentage of that took 3 years. So, when we get a call we go into an environment and we start looking around to identify where this data breach is happening, we’re seeing evidence that starts 3 years ago; and these organizations did not know that this activity was going on, and the criminals know that, the attackers know that.
We often hear about – actually in the last couple of weeks – we’ve heard a lot about China, we’ve heard a lot about the APT, about how they’re infiltrating various organizations in the United States and maybe other places on the planet. But the mere fact for the organizations that the financially motivated criminals are targeting, and the organizations that have massive amounts of data that criminal organizations are making money from – the vast majority of those victims need not worry about China, they need not worry about the APT. And in fact, many of those organizations are not in the Fortune 100, most of those organizations are not government-related ones, and so we know by correlation of indicators of compromise, the malware that we looked at, the source IP addresses and the other information that we were able to gather throughout our investigations, that many of those attackers come from places all over the world. And in fact, as you see the maps up here, these are the top 9 locations that we identified in our studies (see image above).
So, now when we’d look at the types of methods of attacks that attackers are using, the criminals are using, we hear a lot about zero-days. And in fact, when I was walking around the show floor over the last couple of days, I’ve heard a lot of live demonstrations coming out of very loud speakers as you walk around that talked about zero-days and zero-day attacks. Just like everybody else, I’ll admit those are extremely dangerous. The fact that security researchers can go and find zero-days and turn around, sell them to a vulnerability broker, who will then in turn sell them to a government or a corporation or a criminal group – that’s extremely dangerous.
What we thought was pretty interesting is that when we looked at the investigations that we performed last year, not a single one used zero-day in order to get into an environment. Why? Well, the criminals don’t have to burn zero-days in order to get into the vast majority of the businesses in the world. They use the types of attacks that you’re seeing here (see right-hand image): remote access, they literally just log in to the environment via the remote access mechanisms that the organizations have for their own employees. SQL injection – that’s been on the OWASP Top 10 for years, that’s still a problem for e-commerce and web applications. And then the third one there, legitimate account access – that’s exactly what it sounds like. The criminals didn’t use any tools, any special techniques; they literally just logged in to the organizations. And then the fourth one, we have web-based attacks – those are client-side attacks that we’ve been seeing on the rise, using vulnerabilities and things like Java, browser vulnerabilities, or even Flash vulnerabilities.
There are some of the other examples here that we’ll talk about, but really what I want to do is talk closely about the process. Now, these are businesses. These criminal groups run businesses, and the criminal gangs and the attackers are not just a small band of people that are sitting together in a warehouse or a basement someplace. And in fact, there are correlations of all the various investigations; we actually tie the investigations back to 6 different cyber gangs. These 6 different cyber gangs run their operations just like a legitimate business; they follow legitimate processes and have very robust procedures in order to infiltrate their victims, extract the data that they need, and then make money from doing so.
So, the first step in this process starts with greed, starts with money. Many of the members of these criminal gangs have advanced degrees; they have computer science degrees, computer engineering degrees, but they live in countries where the economy is horrible, where they’re not able to get jobs in the field that they actually have their degrees in. So they go and they join various organizations so that they can actually make money, so that they can buy cars, they can buy clothes, they can buy fancy homes. And they’re drawn to these organizations because that’s the only way they can make a good living.
The next phase is victim identification. We talked about how the criminals can go out and find various victims very easily; there’s basically a sea of opportunity for them, there’s no shortage of victims. They literally can identify victims and put them in their target list every single day, and for the rest of their lives they would never run out of victims in order to fuel their businesses.
So, once they identify a victim and they find a victim that they want to compromise, they basically go and infiltrate them. And they use different mechanisms in order to infiltrate, the methods of attacks that we’ve talked about. They’re not burning zero-days in order to get into these organizations. They’re basically infiltrating them via low-hanging fruit type means, and once they get access, they’re diving deeper.
We used to see about 5 or 6 years ago that organizations would get infiltrated and the attackers would find a database, or find a flat file, they would grab it, and they would leave, they’d never come back again. But today we see a great deal of persistency: once the attackers get into your environment, they want to propagate to other areas of the organization, they want to find various data sources, they want to latch on to those data sources and be there for a very long period of time, because the more data they can harvest out of your organization the more money they’ll make. You become, basically, a natural resource to them.
Once they’re inside the environment and they’ve propagated, now they’re going to start aggregating data. We often see the criminals being very bold. As you can see in these icons here, they’re gathering data from various systems in your environment, but they’re not leaving your environment with it, at least at first. They’re extremely bold: they compromise the server, your data center or workstation someplace, and that will be the aggregation point, that will be the place where they take all the data, they dump all the data there and actually keep it there until they’re ready to exfiltrate it. We actually had a case in the last 18 months or so where an organization was infiltrated and the data was being stored on the server, it was accumulating very rapidly, and the system was running out of disk space. Well, bells and whistles went off and the IT folks came by, checked up the system and deemed that it was running low on disk space, so they added a new hard drive to the system. And that process happened 2 or 3 times until someone actually realized: there’s something going on here, and they asked for some assistance.
But once they actually aggregate that data, now they’re exfiltrating it. And they’re not just sending it off in the clear at random points in time, they’re basically using advanced techniques to encrypt that data just like all of us encrypt our customer data; they encrypt it so that when they send it out to systems that they may not have really good hold on – maybe it’s another system they compromised – so it won’t be intercepted. They are very keen on protecting their loot. When they gather that information they want to protect that information as much as they can so that they can make as much money as they can when they go to sell it.
Now, the next piece: when you want to sell it you have to find buyers. You may think it must be hard; if you have a whole pile of confidential data about individuals, or bank account information, it might be a little bit hard to find people who are willing to buy, people you can trust. Well, in these criminal groups they have web forums they all participate in. Inside these web forums there aren’t just 50 members, there’s thousands of members, sometimes there’s up to 10 thousand members, people there that are able to buy and sell stolen data and entrust the relationships in order to do so.
Once they have identified those buyers, they want to liquidate as quickly as possible, because as long as that data sits in possession, the value then decreases. And so they want to liquidate it, collect the cash and start over again.
When they’ve made their money, and that may be millions of dollars off of these attacks, they’re not going off and buying an island someplace, because this is their livelihood, this is their business, this is what they do every single day. So they’re going off and recycling it. They’re using the tools and techniques that they learned to re-implement those against new victims.
Now what I want to do is change gears a little bit. We talked about the major methods of attack, and so I want to share with you a little bit of an attack demo. In this demo, I’m going to show you this attack in two different ways. We talked about the web-based attacks, and this is something we’ve seen growing over the last several years, where when organizations are infiltrated they’re basically getting attacked via, say, a ‘watering hole’ attack, or a targeted attack against someone they know within the organization who has access to information.
So, what I want to show you – you hear a lot about these types of attacks but many of us haven’t really seen what they actually look like in action. I’m going to show you two different flavors of it that actually have the exact same end result and the setup is the same, but it’s using two different types of exploits. So, what we have here is Work directory (see right-hand image); pretend this my computer here, I have procurement documents and some other confidential files, e.g. My Passwords.txt – really secure method of storing my passwords.
But I also have this evil site, and this is not something you normally would see on your computer, but for illustration purposes this is an ftp site out there, someplace out on the Internet, that’s being controlled by the attacker. So, now I’m going to launch my web browser, and what’s the place that most people go when they first launch their web browser? Facebook, right? So, I go to facebook.com, and as you see here’s my profile, you can see some advertisement here like Amazon UK. Oh wait, there’s a friend of mine, Moshiko Davidi, saying that he just found a new job at Fox News. Wow, that’s great! Let me check out what he’s going to be doing there. So I go and I click on the link. Oh, it says “Internet Explorer has encountered a problem” (see image above), so what do I do? “Send Error Report” or “Don’t Send” – not going to send it.
What you see here is, basically, an executable got dropped on my Desktop and the browser crashed. This should raise some red flags for somebody who is a little bit security-savvy, but for a lot of people that would not really raise any red flags at all, they would just move on and maybe re-launch the browser and try to move on with their day. But what we can see happened is that a ‘goldmine.rar’ file was created (see right-hand image) and sent up to the attacker, up to the attacker’s ftp. So, let me go and see what’s in here. I’m opening the archive, and there’s a directory structure here, and here’s all my files. In fact, there’s even more files here: IE passwords and Firefox passwords were exposed.
We’ve actually investigated some attacks that looked very much like this, this is how they actually started: dropped executables on people’s desktops – and off the attackers went. But that was pretty obvious, right? You saw the browser crash, you saw these files get dropped – pretty obvious. So, let’s look at this in another way.
Basically, what we’re going to see is that was an IE exploit that I just showed you; now I’m going show you what a Java exploit looks like. You’ve heard over the last couple of weeks people say: “There’re lots of problems with Java; uninstall Java from your system.” So, let me show you what that looks like when you’re attacked via Java. So, here I am again, I’m not going to explain everything going on, you just learned about it a few seconds ago.
So, here’s these windows again; now I’m going to launch my browser one more time. Launching IE here, going to Facebook again, and now I’m going to go visit the same site for Moshiko, but it’s going to do something else to me. You see a Java Flash for a second, nothing changed here, my browser is still intact (see left-hand image). In fact, I can go back to Facebook if I want – but same end result, ‘goldmine.rar’ was sent up to the attacker’s computer (image below).
Think about this type of situation when you’re thinking about active defense and when you’re trying to basically stop these type of attacks, really detect them doing real-time analysis of your environment. This attack took place in less than 10 seconds. So, doing real-time analysis, doing real-time defense in your environment is extremely important. We’ve seen these types of attacks happen; in fact it was a #4 attack method we had last year.
Nicholas Percoco: So, now what we’re going to do is we’re going to bring out a special guest, somebody I’ve been working with for a long period of time, my team works with him in his organization very often, working hand in hand in cybercrime investigations. So, please join me in welcoming Erik Rasmussen. He’s a special agent in the Criminal Investigative Division from United States Secret Service. Welcome Erik!
So, in this portion of the presentation, basically, what I decided to do is crowdsource, some of the questions I’m going to ask you today. I went out on Twitter a couple of weeks ago, and my Twitter followers and also people who are following #RSAC asked a question. So I’m going to show the question here, and what I wanted to do is, basically, if you can ask a US federal cybercrime investigator any question – what would it be? So we went through, we had about 40 different responses and picked the best 5 or 6 of those.
Here’s the first one from Johnny Cocaine: “Why won’t you guys leave me alone?!?”
Erik Rasmussen: Pass…
Nicholas Percoco: You pass? Ok, let’s go to the next question. So, we have Dan Gleebits. He asked, basically, why the number of cyber investigators is not equal to the number of investigators. I guess he talked to someone in New York City and they told him there’re thousands of investigators, but very low cyber investigators.
Erik Rasmussen: That really stems, I think, sometimes from just not being sure about what the government, especially law enforcement, might be doing to combat problems. Since you hear about cyber in the news you might wonder, okay, shouldn’t everybody worry about that? And then of course as you dig a little deeper you realize that we in the law enforcement have other missions as well, including protective mission which is a huge part of what we do. So it just comes down to division of labor. We have to look at certain projects, certain investigative missions because of our own charter, and we can’t always look at the thing that people think we should be looking at all the time. We have several hundred agents among our thousands that are trained to investigate cyber, but they also cross-pollinate – look at white-collar crime and threats against the President. So, really, you just got to realize that not everybody can do that all the time because we have other things that we need to worry about. People are always asking for more agents to train for cyber. So if one day they do that and the next day they do cyber, then go back to protect the President – it’s getting covered at some point during that cycle.
Nicholas Percoco: Ok, great, thank you! The next question here is from KerenE. This person said they would love to know what’s motivating to keep on fighting the good fight despite immeasurable odds.
Erik Rasmussen: I don’t know if I agree with “immeasurable odds”. It’s a little cynical for me, but I think the best way to describe that feeling for us is: “You’re #1 in public service”, so we serve the country, we serve our federal government, we serve our states, we serve our cities. And it’s our duty to do something like that, just like being in the military. So despite whatever odds might be against us, when you go to the 22 thousand people around here in the RSA Conference, somebody needs to know that we’re working on the cyber problem. If everybody’s seen “Zero Dark Thirty”, there’s a great line in there where it’s talking about who’s looking at Osama bin Laden; and the people in the room look around each other and realize: “Look, we’re the only ones doing it.” Nobody else has got the time to do it or decided this was going to be their mission. And I think, obviously, a select few of us inside the US Government and of course the local governments are now realizing we need to be active in this role. And it ends up being something that people think is immeasurable, but then you realize that tens of thousands of people that are focusing on it in private sector, focusing on it in research and development sectors, and then of course law enforcement – you realize that there’s actually a lot more people, the army is a lot bigger than maybe previously thought.
Nicholas Percoco: Great, thank you! So here’s the next question: what data do you wish the victim had regularly collected prior to calling you guys in?
Erik Rasmussen: Very good question, especially with what you were talking about at the beginning of the presentation. I think because of the issues of timely detection and mitigation, it’s always good to have a victim be very prepared when somebody from law enforcement comes in at the initial stages of the investigation. And of course, you know, there’s sort of a shell shock problem due to the initial confusion that would occur when you’re getting attacked. But especially for Secret Service investigations, which involve lots of financial crime and theft, knowing what was normal activity and what was abnormal activity through your own internal risk assessments, through your own system administrators and how they keep up the network – the kind of information like that is extremely important.
When it comes to logs, people say “network logs” or “netflow”, but what does that mean? We are extremely interested in what gets inside the network. Obviously there’s all sorts of interesting network traffic that never gets inside of the network, and that’s obviously a lot of data. So, what we care about is what’s got in on the normal ports, because it’s very clear that the attacks are occurring on normal, open, usable ports that you have to have; and then also, let’s say, for instance, what’s an unusual port to be inside of your system? So, those kinds of things are important.
And, you know, if you’re willing to share the time before legal process sets in through subpoena or search warrants, have you ever had this kind of a problem before? If you look at that slide that you had up there for the types of attacks – SQL injection, some other sort of exploit – then we can kind of compare and see, okay, this company has had 17 SQL injections in the last 17 months; what does that mean? And all of a sudden this other attack occurs. That’s very interesting to know before you start your investigation.
Nicholas Percoco: Ok, great! We have another question here. This talks about the definition of cybercrime which typically varies depending on who you ask: since definition of cybercrime may not be obvious, where do you stop with crimes that aren’t 100% computer-related, for example phone scams?
Erik Rasmussen: Well, for the Secret Service, I guess you can delineate it between our investigative role and our forensic role. Our forensic role, federally, through our Electronic Crimes Task Forces, of which one is here in San Francisco as well, and through our actual working groups – we have all sorts of tools where a computer might be used in a robbery or homicide, or a cell phone, and our forensic investigators need to be a part of that investigation because of the tools. In investigative side of things, based on the core violations that the Secret Service investigates, well, the computer has to somehow be a central part of that investigation. With all the different definitions out there – you know, there has to be a server that retained all the personally-identifiable information; there has to be the attacking server; there has to be a laptop used by the suspect – they play sort of an integral role. And just the mere fact that some sort of electronic device, of which now there are thousands, is involved, doesn’t mean necessarily trigger a Secret Service investigation.
Nicholas Percoco: I think this is the last question here. Daya asks: “How much felony crime is undertaken using stolen identities?”
Erik Rasmussen: That’s a very tough question to answer when it comes to statistics. But under what we can do federally, you know, we have the United States Code which has all the different federal statutes, and of course there are various data breach laws that states have enacted, including some that are either on the books or being proposed, such as in California. Wire fraud, computer fraud, of course there is an Identity Theft Statute: USC – Section 1028 and USC – Section 1028A, which is aggravated identity theft – those are two of our most commonly used statutes that involve the theft or use of stolen information. And then you have mail fraud of course, this is another commonly used one; and then of course, if you’re working with the local investigator, any of the state equivalents that might be out there: you know, various forgery charges in states are things that we use to investigate the types of felony crime that would be part of our charter.
Nicholas Percoco: Now I’d like to have you spend a couple of minutes talking about some of the successes your organization has had. We spoke a lot about “here’s what the criminals do, here’s what their activities are in terms of infiltrating organizations and stealing data.” Maybe you could talk about some of the results of your investigations?
Erik Rasmussen: Ok. With this great sort of modified lineup of people we have here (see right-hand image), all these individuals were under investigation by the Secret Service due to various computer-related crimes. In the lower right there’s a very interesting picture of Albert Gonzalez with his DEFCON badge around his neck, but he was one of the main hackers responsible for the TJX breach several years ago, and he’s in federal prison now.
There’re some other individuals on there that were involved in a very significant point-of-sale breach that occurred in the food & beverage industry several years ago, that have been recently brought to justice, including an actual Romanian who Romania actually allowed us to extradite to the United States to serve justice in front of the United States Justice System.
There’s an individual on there who was responsible for a denial-of-service attack against a domain name registration company that locked out his domain once they knew it was resolving to a botnet he was running. That was prosecuted out of the Los Angeles office for the Secret Service.
There’s some individuals on there, too, responsible for the very beginning actually of, I guess you could call, some of the Anonymous cases out there. I don’t know if everybody recalls, but in about 2006 or early 2007 there was an individual that was responsible for a denial-of-service attack against the Church of Scientology after a very infamous Tom Cruise video surfaced on YouTube. And he was one of the individuals responsible for initiating that attack.
And then we have also on here more traditional white-collar crime where an individual worked at a telecom kiosk in a mall, and he and his girlfriend used the ability to have a 100% discount for the cell phones at their store, give them to himself via various post mailboxes, then resell them on eBay and to other third parties, where he was defrauding that store for about $1 million with the cell phones, from a single store.
So, as you can see, all those crimes may be things related to what’s in the Global Security Report or other investigations that are very much related to things that the Secret Service does.
Nicholas Percoco: Ok, Erik. So, I guess one of the big questions that people often ask us is, you know, why are we working together? Why are we here sitting on stage together? You often hear a lot about public-private partnerships; this is a perfect example of one that actually works and that’s actually been successful. We’ve done a lot of investigations, we’ve shared a lot of results of those investigations with Erik’s organization, with the Secret Service, and internally gained a lot of knowledge about the criminal activity, the criminals behind the activity. And that helps us in our investigations to understand better how to investigate, better understand the mindset of where they’re going in those organizations, what they’re targeting. And so, maybe you could add a little bit – what benefit do you have by working with an organization like Trustwave?
Erik Rasmussen: The Secret Service is extremely proud of that relationship, and I think what it builds is a level of trust with the rest of the private sector and with other government sectors that may at some point need the resource of Trustwave and/or the Secret Service. When they see people collaborating and being able to work and share information – because I think information sharing maybe is an overused term – that’s important because people of course want to be able to assist in an investigation, maybe get some sort of feedback before, during or after. And that occurs with us, especially with our point-of-sale investigations. The incident response side of Trustwave is extremely robust and it shares a lot of information with the Secret Service and vice versa. And also, I think we help each other out with various training modules that we can send out to the different Secret Service entities to be able to show that both sides can learn from what it’s like to be able to experience the cyber environment privately and the cyber environment in the government sector.
Nicholas Percoco: Great, thank you! So with that, there’re two things that you take away from this session, or we hope you take away. One is: better insight into the world of the financially motivated cybercriminals. And the second piece is that public-private partnerships do work, and that’s by sharing information with law enforcement, by engaging with any organization involved in criminal investigations. Folks in the security industry can really help put a really big dent into the criminal enterprise, into the cybercrime that’s going on today.
So with that, thank you Erik! And everybody here, I’d like to thank you for listening!
