John Strand, the owner of Black Hills Information Security, shares his perspective upon what the present-day penetration testing should be like.
The name of this presentation is “How not to suck at pen testing”. There’s a lot of presentations that you’ll see where people just rip on the pen testing industry, and trust me, we are going to have a couple of slides of that. But what I want to try and establish is some type of mental framework whenever you go in to a pen test, to establish what it is you are going to do, how you are going to do it and, more importantly, why we are doing what we are doing.The first thing I want to get across is I believe that we have a major issue in information security whenever it comes to the area of network penetration testing (see right-hand image). And let me explain why. When most of you do vulnerability assessment scanning today, what is the main requirement that drives your vulnerability assessment scanning for most of your vulnerability assessment scans? PCI, right? How are those reports for you? Whenever you get set up and you do a PCI scan, what do you have to do in the executive report? You have to list every single finding. In the executive report. When we went for PCI ASV certification, our executive report was 550 pages long.
The reason why I bring that up is because we allow that to happen. The idea of a vulnerability assessment was boiled down to running a vulnerability assessment tool, kicking out the report, and giving that as the findings. Guess what? That’s now the standard. Most organizations that I know today, whenever they do PCI ASV scanning, they run the ASV scan, they get the report, and then they promptly throw the damn thing away. And you know what? That’s probably the best move they could possibly do. The only thing that that does is establish some level of due diligence, so some ASV scanner or some security team can say “Hey, we told you guys about the vulnerabilities; it was in that report that was 1000 pages long, so my ass is covered from the scanning perspective.”
I am terrified that what’s ultimately going to happen in the area of network penetration testing is it’s going to be reduced down to some kind of simple concept, some simple methodology that can be automated, and then at that point the wolves will come in and they will say “You know what? We can make a tool that will do this automatically. We will establish a standard of what a network penetration test should be.” And as soon as that happens, every single ounce of creativity that all of these beautiful crazy people do – they no longer get to do that. That’s where we are headed right now. If we allow that to happen, it’s going to be in our future, and then all of a sudden this industry is going to suck. A lot. And then we’ll have another set of issues to deal with.I don’t want to rip too much on scanning (see right-hand image), because I think scanning is a beautiful thing. There’s a lot of talk about what type of vulnerability assessment scanner you should or should not use. A lot of the vendors fight and they argue over it. But almost any penetration tester I know, worth their salt, they don’t look at the scanners as “end-all and be-all” of what should be done and the vulnerabilities that are important to address on the target systems. Instead, we look at it as our eyes and our ears.
I mean, honestly, what was the original definition of a hacker if we go back 20-30 years ago? What was the original definition of a hacker? Was it simply somebody who was maliciously trying to break into computer systems, or was it something different? What was it? Actually, a lot of the original hackers were people that tried to make things actually work. And how did you actually make things work back in the day? You had to know how they worked, right? You had to take them apart, you had to understand them. And that’s what a network penetration test should be. And a vulnerability assessment scan is outstanding for taking that first step and trying to learn as much as we possibly can about the systems that we are being hired to break into, whether or not you are a consultant, or whether or not you are doing it internally as part of your own security team.Unfortunately, whenever many people look at vulnerability assessment scans, what do they do? What is the first thing you do? What’s the first thing you are going to look for in that report? Criticals. And what color are they? Red. So we are going to look for Red (see right-hand image). Unfortunately, today the methodology of simply running a vulnerability assessment scanning engine, looking for the Red and then using Metasploit to try and find those vulnerabilities and then exploit those vulnerabilities, is at least five years out of date. It just doesn’t work that way anymore. We run into this all the time when we work with customers and they have pen test done by other companies – we are just going to call them a generic term “pen test puppy mills”. We have these pen test puppy mills that run the scanner, take the report, convert it into a Word document and submit it to you.
Whenever I’m teaching for the SANS Institute, that’s one of the most common questions I get. They come up to me and they say “I really need a tool that will take Nessus results and will convert it into a Word document format – do you know of a tool that does that?” I’m like “No, you shouldn’t do that.” And they are like “But my boss requires it, my CIO requires it, they want it to be converted into that particular format, they want it sorted, and it has to have our logo on it as well.” That’s ultimately what they are looking for, and they are not actually using that vulnerability scan. They are not taking those results and using them to try to understand their target networks.
Years ago, I did a “capture the flag” and I had a challenge. It was a web server. On this web server, it was just a simple website and it had log and user ID and password. And the students would come into this “capture the flag” and they would fire up Burp, they would fire up Zed Attack Proxy and they would attack it and try to find vulnerabilities, including cross-site scripting vulnerabilities. They would run Nessus, they would run Acunetix, they’d run all of these awesome tools against this particular website. And they wouldn’t get in. Finally, at the end of the night, when no one would ever get that particular flag in the “capture the flag” challenge, people would come up to me and they would say “How do you get that flag on that web server?” I’m like “Did you open a browser and go to the web server?” They’d say “Well, no…” He opened a browser and went to the web server, it said “User ID”, “Admin”, “Password”, whatever the password was at the time.
We relied on the tools, because if the tool didn’t come back with a red vulnerability, then clearly there was no way that we’re ever going to break into this computer system. See, what’s happening whenever we allow that to occur is we are taking the human out of the mix, we are taking the intelligence out of the mix, and it’s being reduced down to an automated process. I’ve said this many-many-many times: if you can be replaced by an automated tool, you will be.
That’s why I feel really-really deeply in my heart anytime I’m on any of the mailing lists, and people, you know, talk about what a pen test should be and I put in my two cents, and they come back to me and they say “My job is converting Nessus scans into Word documents, and that’s what I do every single day.” You are not doing it because it’s what you want to do. You are doing it because you have one day to do an external network assessment for 1000 IP addresses, and somehow you’ve got to get a report and you’ve got to provide it. And your boss rates how quality your work is, based on how many pages you are able to convert from your Nessus scan results. I feel for every single one of you. I really do. If I could pick you all up like stray puppies and hire you, I would. But I already have enough stray puppies. It’s like a litter.So we’ve got to try and move past this looking for Red. The solution is, let’s start looking at the other findings (see right-hand image). Many of the people, when they join Black Hills Information Security, I allow them to make mistakes. It’s a bit strange, and I know it’s probably a bit hard, but I understand what mistakes many people that are new to network penetration testing are going to make. The first mistake that we see is they run a Nessus scan, there are no Reds, and they say there’s nothing here that we can possibly break into, nothing at all. Another mistake is, you get access to a workstation and you say “I’ve got to elevate to root” or “I’ve got to elevate to administrator or system.” And they spend three days doing that. The gentleman that took the floor right before me said “Just pillage, see what you can get with the level of access you have, that’s awesome!” That’s ultimately, once again, going back to the pure definition of what a hacker is. You are using these tools to learn about the environments that you are breaking into.
So use your vulnerability assessment scans as your eyes, if that’s the type of pen test you’re doing. There’s a number of different pen tests that you can do, but if you are doing the standard vulnerability assessment and then trying to exploit computer systems, that’s awesome if you are going to do that. But if you are going to use that approach – those are your eyes, it’s your ears, it’s your sensory input. And you’ve got to almost love the organizations that you are trying to break into, because you want to learn as much as you possibly can about them.