How not to suck at pen testing – John Strand

John Strand, the owner of Black Hills Information Security, shares his perspective upon what the present-day penetration testing should be like.

The name of this presentation is “How not to suck at pen testing”. There’s a lot of presentations that you’ll see where people just rip on the pen testing industry, and trust me, we are going to have a couple of slides of that. But what I want to try and establish is some type of mental framework whenever you go in to a pen test, to establish what it is you are going to do, how you are going to do it and, more importantly, why we are doing what we are doing.

The first thing I want to get across is I believe that we have a major issue in information security whenever it comes to the area of network penetration testing (see right-hand image). And let me explain why. When most of you do vulnerability assessment scanning today, what is the main requirement that drives your vulnerability assessment scanning for most of your vulnerability assessment scans? PCI, right? How are those reports for you? Whenever you get set up and you do a PCI scan, what do you have to do in the executive report? You have to list every single finding. In the executive report. When we went for PCI ASV certification, our executive report was 550 pages long.

The reason why I bring that up is because we allow that to happen. The idea of a vulnerability assessment was boiled down to running a vulnerability assessment tool, kicking out the report, and giving that as the findings. Guess what? That’s now the standard. Most organizations that I know today, whenever they do PCI ASV scanning, they run the ASV scan, they get the report, and then they promptly throw the damn thing away. And you know what? That’s probably the best move they could possibly do. The only thing that that does is establish some level of due diligence, so some ASV scanner or some security team can say “Hey, we told you guys about the vulnerabilities; it was in that report that was 1000 pages long, so my ass is covered from the scanning perspective.”

I am terrified that what’s ultimately going to happen in the area of network penetration testing is it’s going to be reduced down to some kind of simple concept, some simple methodology that can be automated, and then at that point the wolves will come in and they will say “You know what? We can make a tool that will do this automatically. We will establish a standard of what a network penetration test should be.” And as soon as that happens, every single ounce of creativity that all of these beautiful crazy people do – they no longer get to do that. That’s where we are headed right now. If we allow that to happen, it’s going to be in our future, and then all of a sudden this industry is going to suck. A lot. And then we’ll have another set of issues to deal with.

I don’t want to rip too much on scanning (see right-hand image), because I think scanning is a beautiful thing. There’s a lot of talk about what type of vulnerability assessment scanner you should or should not use. A lot of the vendors fight and they argue over it. But almost any penetration tester I know, worth their salt, they don’t look at the scanners as “end-all and be-all” of what should be done and the vulnerabilities that are important to address on the target systems. Instead, we look at it as our eyes and our ears.

I mean, honestly, what was the original definition of a hacker if we go back 20-30 years ago? What was the original definition of a hacker? Was it simply somebody who was maliciously trying to break into computer systems, or was it something different? What was it? Actually, a lot of the original hackers were people that tried to make things actually work. And how did you actually make things work back in the day? You had to know how they worked, right? You had to take them apart, you had to understand them. And that’s what a network penetration test should be. And a vulnerability assessment scan is outstanding for taking that first step and trying to learn as much as we possibly can about the systems that we are being hired to break into, whether or not you are a consultant, or whether or not you are doing it internally as part of your own security team.

Unfortunately, whenever many people look at vulnerability assessment scans, what do they do? What is the first thing you do? What’s the first thing you are going to look for in that report? Criticals. And what color are they? Red. So we are going to look for Red (see right-hand image). Unfortunately, today the methodology of simply running a vulnerability assessment scanning engine, looking for the Red and then using Metasploit to try and find those vulnerabilities and then exploit those vulnerabilities, is at least five years out of date. It just doesn’t work that way anymore. We run into this all the time when we work with customers and they have pen test done by other companies – we are just going to call them a generic term “pen test puppy mills”. We have these pen test puppy mills that run the scanner, take the report, convert it into a Word document and submit it to you.

Whenever I’m teaching for the SANS Institute, that’s one of the most common questions I get. They come up to me and they say “I really need a tool that will take Nessus results and will convert it into a Word document format – do you know of a tool that does that?” I’m like “No, you shouldn’t do that.” And they are like “But my boss requires it, my CIO requires it, they want it to be converted into that particular format, they want it sorted, and it has to have our logo on it as well.” That’s ultimately what they are looking for, and they are not actually using that vulnerability scan. They are not taking those results and using them to try to understand their target networks.

Years ago, I did a “capture the flag” and I had a challenge. It was a web server. On this web server, it was just a simple website and it had log and user ID and password. And the students would come into this “capture the flag” and they would fire up Burp, they would fire up Zed Attack Proxy and they would attack it and try to find vulnerabilities, including cross-site scripting vulnerabilities. They would run Nessus, they would run Acunetix, they’d run all of these awesome tools against this particular website. And they wouldn’t get in. Finally, at the end of the night, when no one would ever get that particular flag in the “capture the flag” challenge, people would come up to me and they would say “How do you get that flag on that web server?” I’m like “Did you open a browser and go to the web server?” They’d say “Well, no…” He opened a browser and went to the web server, it said “User ID”, “Admin”, “Password”, whatever the password was at the time.

We relied on the tools, because if the tool didn’t come back with a red vulnerability, then clearly there was no way that we’re ever going to break into this computer system. See, what’s happening whenever we allow that to occur is we are taking the human out of the mix, we are taking the intelligence out of the mix, and it’s being reduced down to an automated process. I’ve said this many-many-many times: if you can be replaced by an automated tool, you will be.

That’s why I feel really-really deeply in my heart anytime I’m on any of the mailing lists, and people, you know, talk about what a pen test should be and I put in my two cents, and they come back to me and they say “My job is converting Nessus scans into Word documents, and that’s what I do every single day.” You are not doing it because it’s what you want to do. You are doing it because you have one day to do an external network assessment for 1000 IP addresses, and somehow you’ve got to get a report and you’ve got to provide it. And your boss rates how quality your work is, based on how many pages you are able to convert from your Nessus scan results. I feel for every single one of you. I really do. If I could pick you all up like stray puppies and hire you, I would. But I already have enough stray puppies. It’s like a litter.

So we’ve got to try and move past this looking for Red. The solution is, let’s start looking at the other findings (see right-hand image). Many of the people, when they join Black Hills Information Security, I allow them to make mistakes. It’s a bit strange, and I know it’s probably a bit hard, but I understand what mistakes many people that are new to network penetration testing are going to make. The first mistake that we see is they run a Nessus scan, there are no Reds, and they say there’s nothing here that we can possibly break into, nothing at all. Another mistake is, you get access to a workstation and you say “I’ve got to elevate to root” or “I’ve got to elevate to administrator or system.” And they spend three days doing that. The gentleman that took the floor right before me said “Just pillage, see what you can get with the level of access you have, that’s awesome!” That’s ultimately, once again, going back to the pure definition of what a hacker is. You are using these tools to learn about the environments that you are breaking into.

So use your vulnerability assessment scans as your eyes, if that’s the type of pen test you’re doing. There’s a number of different pen tests that you can do, but if you are doing the standard vulnerability assessment and then trying to exploit computer systems, that’s awesome if you are going to do that. But if you are going to use that approach – those are your eyes, it’s your ears, it’s your sensory input. And you’ve got to almost love the organizations that you are trying to break into, because you want to learn as much as you possibly can about them.

A number of years ago Ethan Robish, who was with Black Hills Information Security, was doing a pen test for a customer of ours that had multiple pen tests from “pen test puppy mills”, that was being pen-tested about every single quarter. And the pen test puppy mills did the same thing: they ran the scan, they printed out the report, they handed the report in, and they said “You know what? You guys are great, there’s no Criticals, there’s no Reds. Completely ignore all the mediums, lows and informationals, because who cares about those?”

So Ethan started going through the findings and he saw something that caught his attention. It was RoboCop riding a unicorn (see right-hand image). No, that’s not what he found…

He saw a website with the name ‘hello.php’ (see left-hand image). Now, its name is not an evil backdoor name at all, it was just innocuously called ‘hello.php’. And he thought “What the heck, I want to know what ‘hello.php’ is.” So he surfed to it and he saw this. I’m going to give you guys a couple of tips. Number one: if you’re on a pen test and you go to a website, and there’s a skull and crossbones in the upper left-hand corner – that’s generally a bad site. Tip number two: if you’re doing a pen test in 2013-2014, and the skull and crossbones right next to it has ‘2009’ – that’s also a bad site.

What this was is a PHP backdoor. How many pen testers here love WordPress? I just see the guys sitting and smoking pot, going “You know what we need? We need a way that people can upload crap to our servers without authenticating. Dude, make that happen!” And for years pen testers rejoiced at this capability, where somebody had uploaded a backdoor to the server and it had been compromised for years. For years. And it was hard, because when we went to the customer we said “Hey, we found this critical backdoor on one of your servers,” – they said “Well, was it Red or Purple in the report?” That’s what we’ve been trained to look for, right? It’s like Pavlov’s dog – ting, it’s time for steak! We need to move past that, because this is wrong, and this is the type of stuff that people are paying you to do, whether or not you’re an external consultant or you’re doing it internally for an organization.

Another one (see right-hand image). A couple of nights ago, or actually a couple of months ago – time flies – we were doing a network pen test. This is a company that we had tested quite regularly, and we were really just slugging through these informational and low warnings, and there’s just thousands of them. One of my favorites, directory indexing, is a goldmine. Do yourself a big favor – if you can go into your Nexpose, your Nessus configuration, any finding that has ‘share indexable directory’, anything like that – crank that puppy up the high so that every time it shows up, it is bright-red critical and you look at it.

This was a customer that, I guess, had a lot of other customers, and they sent out a lot of emails on behalf of those customers. You think MailChimp, but it’s not them. Anytime there was an error, the application would actually drop a log file into the temp directory. And the log file would name out all of the customer data: the user IDs, their passwords, all of the good stuff that you would ultimately look for in a pen test. And their application was generating error logs. If you are in the pen testing field, error logs are your best friend, especially for custom applications, because they bleed all kinds of horrible things.

As you can see, we had hundreds of these CSV files that would get to a certain size and then they would roll over within a certain time period, and it was just riddles with personally identifiable information, user IDs and passwords for all of their customers. Not unlike some of those major breaches that have happened over the past couple of months, this would have been public, this would have been one of those massive breaches that would have shown up on the front page of USA Today. This was not Red, this was not Purple, there was no leet hacking, there was no 0day involved in this – it was just an issue of treating your scan results, treating your data as your eyes to try to learn as much as you possibly can about a network.

Another thing that we like to play with – sometimes you don’t really have anything. It says “Oh, there’s SMTP server found.” I love notifications and warnings like “We found SMTP server”; or another one of my favorites is “We found a Telnet server”. Now, what would you ultimately do if you find a Telnet server open on the outside of the Internet? You’re going to go to it, right? Take the banner, then do a lookup – what type of device it is – try the default user IDs and passwords, and nine times out of ten you are going to get in with those.

But this (see right-hand image above) was neat. This is one that was done by Brian. He found out that you could connect into their SMTP server. And if you try to send an email to an email outside of that SMTP server, it would pop up an error and say “Sorry, that domain isn’t in my list of allowed recipient hosts”. However, he discovered that if you send an email, while logging directly into SMTP server, to an internal email address from an internal email address, it would say, oh, that’s fine, you can perfectly do that. No problems whatsoever, completely bypassing all spam filters, completely bypassing any other filters.

Now, why would this be configured this way? This is an easy configuration change to make it not do this. But what are people scanning for? What’s the biggest worry you would have with SMTP server exposed on the open Internet? Relay, right? This wasn’t a relay server, because I couldn’t send emails outside of that organization. But you could spam the inside of the organization all day long, with absolutely no problems whatsoever.

A couple of years ago, we had something very similar, where we could access their internal SMTP server from a compromised workstation, and I could shoot email to anybody from the organization from anywhere I wanted it to be. So I shot an email that was titled “TEST: Sale!! We are going bankrupt!!!!!! Everything must go!!!” from Abraham Lincoln, thepresident@whitehouse.gov. In it, it said “Theater sucks.” And I remember the customer getting really-really frustrated with me and saying “That’s just in poor taste!” And I said to the customer “Too soon..?”

This also requires us, as network penetration testers, to go beyond the scanning results (see right-hand image). Somebody earlier today was demonstrating how you could harvest anybody’s Social Security Number. That’s awesome! Do you think the bad guys will use that against you? Absolutely! That’s outside of the bounds of how normal pen testing is done. They run a vulnerability assessment scanner, they keep just to that vulnerability assessment scanner results. We have to go beyond the scanning results. Everything is not going to come in a standard Excel spreadsheet that you need to deal with.

Another kind of offset one was antivirus. I know that this isn’t leet at all, but a couple of weeks ago we did a webcast called “Sacred Cash Cow Tipping” on bypassing AV engines, and we bypassed all of the AV engines that are up here (see right-hand image). It was an awesome, pure technical challenge, and it wasn’t all that difficult. In fact, I shot an email to most of the guys the night before and said “Hey, did you bypass Symantec?” And most of them were like “Oh, crap, I got to sober up for a little while to bypass the AV. I’ve been drinking wine with my wife for the past two hours.” But it shouldn’t be that big of a deal, and we were able to bypass all of them relatively easily. There wasn’t much difficulty in bypassing them.

Why does this matter and how does it tie into this presentation? I say this in a lot of presentations: computer security is like architecture and structural engineering. And I want to explain exactly what that means. My wife is a structural engineer, and I like to read her books. And everything in structural engineering I’ve noticed is all about failure, and she gets really uncomfortable when I bring that up. Everything in her books is about failure. What is the failure point for a beam spanning this side of the room to the other side of the room? What is the failure point for a column in the corner of the building? What is the failure point, believe it or not, for doors: if there is some type of failure, how much weight can doors hold before they collapse?

That’s structural engineering. And think about what that’s based upon. That’s based upon thousands of years of jackasses breaking things. We are those jackasses. That is our calling. We are the people that are finding the flaws in components, the different components that make up our architectures today. We are the ones that are going through and finding those failure points. And the reason why this is important is because we need to demonstrate to management that yes, Symantec can be bypassed; yes, McAfee can be bypassed; in fact, all of them can be bypassed. This is a structural weakness for AV. It’s not about finding the right antivirus solution, it’s about mitigating that structural weakness.

There’s not a single component in this building that is completely impervious to any type of load that you would put on it. Everything fails at certain points. When we are pen testing, we are finding those failure points in those components. And then we need to share that with our customers, we need to share that with the other people within our organization so that they better understand the limitations of the materials they are working with in their IT infrastructure. That’s why we do this. It’s more in line to show people what is possible and then try to move them to a better place to start thinking about how they are going to mitigate these overall vulnerabilities.

We have the slides and we have the presentations (see right-hand image). We couldn’t do all the AV engines in one presentation, so everybody on the team created a slide deck and videos demonstrating that. The reason why we created separate slide decks and separate videos demonstrating bypass live was because I really don’t want to get sued. It’s not something that’s really high on my list of things to accomplish. I call it a “Mitnick complex”. I love his comment today: “I don’t want to go back to prison.” I don’t want to get sued, it’s not something that I want to do. But we went through and we created all these videos so that, if somebody did come back and they’d say “You didn’t bypass our AV,” we can actually prove it.

What else can we do? Well, stepping beyond the organization, once again, going back to Kevin Mitnick and Dave Kennedy, Dave talked about Jigsaw. And what was Jigsaw used for? Harvesting contacts, right? And Recon-ng is a great tool written by Tim Tomes that has a number of modules for harvesting contacts. One of the things that we like to do is harvest the contacts from an organization and then use services like ‘PwnedList’, ‘Have I been pwned?’, ‘BreachAlarm’, and then, basically, check to see if any of those email addresses associated with the target organization have been compromised on third-party sites where the attackers post all the user IDs and passwords that they have harvested (see left-hand image above).

So the question is: how bad can it be? It can actually be very-very bad (see right-hand image). These are some that I pulled a month ago: rand.org, nsa.gov; disney.com has 10308; Northrop Grumman has 6607 – these are user IDs and passwords that are floating around on the open Internet. I was giving this presentation, and I used to do live demos. You should never do live demos. It doesn’t matter how smart you are, the demo gods are going to kill you. I was doing a live demo and I actually showed some information of a company, and somebody came up and they said “You know, that’s kind of a gray area that technically may be illegal with you showing user IDs and passwords of organizations.”

So, I pick on Circuit City now. It’s like “This guy really hates Circuit City, he hacks them all the time…” But you can see what we get (see left-hand image). We can see the actual email addresses associated with the Circuit City employees, who are now unfortunately all terminated. We can also see their passwords, because many websites store those user IDs and passwords. Is your Nessus or Nexpose scan going to actually show you this vulnerability? Never. This requires it to go outside of the bounds, to start thinking beyond what it is that we’re getting from just our scanning results and trying to go deeper.

I also want to spend a few seconds and talk a little bit about getting caught (see right-hand image). It was discussed earlier why getting caught is important, and I want to kind of help establish a couple of frameworks and some examples. This is something that we’ve been doing at BHIS for a long time, and we strive to get caught, but a lot of times we treat it more scientific. We have an actual Excel spreadsheet, we show the different types of malware, the different types of activities that we did. And the idea is to find that clipping level – where can they actually detect you – and then give them a gap analysis on what they can do to get to the next level so that they can, basically, start detecting the more advanced attacks. I’m really sick and tired of pen tests where pen testers are, basically, saying “Hey, we hacked you, we’ve got domain admin and we’re done.” That’s not a pen test. That’s just you trying to prove that you’re awesome, and it usually comes across as a jerk. We don’t want to do that.

Instead, what we want to do is something like this (see left-hand image), and this is actually probably one of the better organizations that we tested. This is just a small section of the spreadsheet. You generate a whole bunch of different types of malware – custom malware, ascii shell code injection, encoded malware, raw malware – a number of different ways of trying to get it to the target organization, and you simply record. Did they detect it? Yes or no. Fail or pass. So, then whenever you go to the organization you can say “These are the holes in your overall security architecture, and these are the things you can do to try to mitigate these overall holes.” And that’s great! A kind of side effect and bonus of this is, if you do screw up and you get caught you can be like “That’s awesome, we’re going to put that in the Excel spreadsheet, you guys did good.” It’s like, oops, I really shouldn’t have sent that unencoded Meterpreter into the organization.

Also, there’s data loss prevention. As I said, we’re in the midst of a webcast called “Sacred Cash Cow Tipping”. In information security, there’s a lot of different technologies that are becoming sacred cows. We’ve got AV, it’s clearly a sacred cow. Even though most people in here can bypass it in a matter of moments, most organizations continue to pin their entire security hopes and dreams upon AV catching things. But if AV doesn’t catch it, that’s great – we can use data loss prevention. Has anybody ever been in a pen test and DLP completely shut them down? If so, you don’t want to raise your hand. Data loss prevention, as near as I can tell, is only used for keeping honest people honest and helping catch people that make general mistakes. For a real attacker, it never works.

How would you know of that? Because if you were listening to the product marketing materials that are provided, you would think this is the best stuff since sliced bread, when in fact it doesn’t work at all. Test it. That’s what we are here to do. We are here to break things. That’s what we do – find the structural flaws.

But every time we move forward things get harder, and then we find more vulnerabilities (see left-hand image). And this is a good thing. If you look at group policy preference files – I mean, how many of you have gotten domain admin over group policy preference files in the past four or five months? It’s awesome, right? And how many of you were like “Boy, this new shell vulnerability, this bash thing, this is going to be my meal ticket for the next couple of years”? We kind of get hung up on those things. But slowly we start improving. Slowly we’ll start removing some of those structural vulnerabilities. I’m now joking with my customers and saying “The worst thing you can do for information security today is run Active Directory.” It’s like a super-highway of breaking into organizations.

But we make these steps forward and then we find more vulnerabilities. One of the fun things about Black Hills Information Security is we have a lot of customers that listen to Security Weekly. And they listen to what we recommend, they wait until I get everything kind of squirreled away from the recommendations, and then they call us. They’re like “We’ve done everything that we learned in your class, five of four hacker techniques, we did everything on the show, now you can come and pen test us.” And we’re like “Oh, that’s just great, awesome, this is going to be fun!”

We’ve run into a couple of situations where organizations have implemented Internet whitelisting. I’ve even said on this stage that we need to move to Internet whitelisting, and I still think that that’s the right thing that we need to do. We need to start moving to more whitelisting approaches. Does that mean it’s 100% secure? No. Does that mean it’s better than traditional Internet blacklisting? Absolutely. So we need to try to move people to that. But that doesn’t mean that we are not going to have stumbles along the way.

So, Ethan found out in one of his pen tests that if you applied Internet whitelisting and you did a Google search, let’s say for Google’s blah, it would say you can’t go to that website. What you can do is you can do a search that includes the domain of the organization that you’re trying to break into or out of, and it would say that’s perfectly cool (see right-hand image). The reason why is because a regular expression goes through the URL and it looks for whitelisted domains. So, let’s say that the target organization is hacked.com.

If you can do a Google search for hacked.com, it sees “hacked.com” in that URL and it allows you to bypass it, and it allows you to go to Google. So if you have a backdoor, you can have the backdoor communicate over HTTP, and all you need to do is put in a variable – in this situation, it was “blah” – equals (=) “hacked.com”, or target company dot com, and it allows you to bypass it (see left-hand image).

Does this work on different organizations like Websense?

Yes, this works on Websense as well (see left-hand image). So, if you have an organization and you have to get out of that organization, all you need to do is gen up your backdoor to make sure it has the target URL of the organization – and it’s going to let you through, because that is an authorized domain. What’s really cute is whenever you try to give this to the vendors and say “Look, this is something you might want to fix,” what usually happens with vendors is they say “Well, yeah, that’s the way regular expressions are supposed to work.” And you’re like “Really?!” They’ll say “Yeah, this is a feature.” No, it’s not. But slowly, people get better.

The next two technologies I’m going to talk about – I don’t want to rip on them, because it’s going to sound like I am – but I actually love both of these technologies quite a bit. The first one is Bit9. I love Bit9, but is it a pain to install? Absolutely. But we do network pen testing and we have to bypass Bit9. Now, bypassing Bit9, for most ways to bypass it, isn’t actually trying to bypass Bit9 itself. What you are really trying to do is find configuration weaknesses in how they actually configured it. And usually the way that this is done is trying to find a file type that is authorized in that organization.

One of my favorite ways for bypassing Bit9 is PowerShell scripts (see left-hand image). Why would PowerShell scripts be allowed to bypass Bit9? How many of your organizations use PowerShell as part of their day-to-day activities? So, a lot of what we are doing is we are trying to find those gray areas, where we can exploit those gray areas in the efforts of trying to blend in. Mr. Mick Douglas is going to give a presentation a little bit later on about Powercat, which is a PowerShell implementation of Netcat. All you need to do is put your malware or have it download, and you can do direct memory injection using PowerShell, and it will bypass most implementations of Bit9. You can actually do digital code signing certificates for your scripts.

You can do a lot of things to try to mitigate this but, by and large, most organizations will allow all .ps1 files to execute, or they’ll allow all JAR files to execute. Why? Well, we need to allow Java. This is jrDesktop (see right-hand image), which is just a very nice implementation of VNC in Java, and almost every single organization we have tested will allow you to fire up a backdoor using jrDesktop. This isn’t leet hacking, it’s just trying to find those gray areas that most organizations miss whenever they are trying to establish their security architecture.

The other thing is ISR Evilgrade attacks (see left-hand image). The reason why this works is because of something called “inheritance”. Whenever you have an application that runs, many times you will say “I trust this application” or “I trust this directory that this particular application executes from.” And then anything else that executes in that directory, or anything else that is executed from an executable that is already trusted, will be trusted as well. So, if you have Java updater, Java updater is trusted because you want to update your Java. When Java updater goes out and pulls down an executable, you can give them malware using ISR Evilgrade, and then that malware will be trusted because it was started from Java updater. So you, basically, take advantage of that issue of inheritance. And this is probably one of the more difficult things for organizations to tune Bit9 to actually get it to work properly to detect it.

Palo Alto and their application inspection – once again, I like this product. I think these guys are a great vendor, they do really cool things. Once again, is the product super-easy to implement? No. It takes a lot of love, care and feeding to try to implement it properly.

One of our favorite tools for bypassing these products, or products like Palo Alto, is Ron Bowes’ dnscat (see left-hand image). How many of you were on Bowes’ talk today? Ron was there, that’s good; although he looked like he wasn’t sure. I don’t quite know why he looked a little bit confused about that. If you are working with a firewall vendor and you want to get a discount on that product, fire up Ron’s tool. Ron gives it away for free, I mean the tool: dnscat is the tool that Ron Bowes gives away for free. I’m trying to be more precise when I speak.

Dnscat is an outstanding product and you can test it, and whenever you get a full shell bypassing someone’s product, all of a sudden the price tends to come down a little bit from a cost perspective. If you ask vendors, they are going to say “Oh yeah, we totally catch that, because dnscat uses Port 53 UDP, and we monitor that port.” No, that’s not how dnscat works. It’s not a shell over TCP or UDP. If that were the case, Ron’s tool would be the lamest tool ever. It does more than that. It does all the backdoor communication over well-formed DNS requests and responses, and that’s what makes it awesome, that’s what makes it very good.

Another thing you can do – we have a tool called VSAgent that we have created; and all of the communication with the tool goes back and forth in well-formed HTML, and we insert all of our command-and-control in a value called “_VIEWSTATE” (see right-hand image). Anybody encounter _VIEWSTATE in their web application pen test? How long can this field get? I think we had a record in one of our tests, for us personally: four pages long of VIEWSTATE data. And the customer is like “Why is it taking you so long to scan our website?” I’m like “Because it’s broken, that’s why!” But this makes an outstanding field for us to exfiltrate data. This makes an outstanding field for command-and-control, because it’s meant to be randomized, right? It’s meant to be a long, variable link, usually very-very long. And most of your firewall tools will not do analysis on those fields because it’s impossible for them to do analysis on every single one of those fields. So, that’s an easy way to try and bypass them.

John Strand: So, let’s talk about trying to find new areas, and that’s kind of where we are going to start tying this up (see right-hand image). We need to find new areas as network pen testers, and we need to continue pushing things as network pen testers beyond just saying “I hacked a toaster.” We need to find additional things that we can do as part of our network security assessments and network penetration tests. Keep pushing it – they will never be able to automate it, they will never be able to create a standard that says “This is what it is,” if we keep pushing the boundaries.

One of the things that hit me is kind of a weird, strange narrative. One of our good friends works at a very large bank. One day he said “Wait a minute, I want to know if we have any banking backdoors or Trojans that are not being actively detected by our current antimalware suites.” That’s kind of a tall order, right? How would you go about doing that? One of the things he noticed, though, whenever he was reverse-engineering some of these really high-end backdoor Trojans, is they hated Zeus. Anytime Zeus hit that box, those backdoors would immediately kill Zeus. Why? Competitor, right? Take out the competitor. You want that data.

So he got an idea that I think is a bit extreme – I don’t know if I would recommend it – but he modified Zeus so that it wouldn’t spread and then deployed it to every single one of his systems. Then he queried “Who has Zeus installed?” Everybody but those five systems – go investigate those five systems right away. We have noticed many situations where malware gets on a system and one of the first things it does is it looks for other malware and it tries to circumvent it, uninstall it and destroy it.

As network pen testers, aren’t we supposed to be trying to emulate what the bad guys do? Couldn’t we do that? I know this is a bit presumptuous to say I think that network pen testers may come up with some creative ways of trying to find malware in environments. Because we create malware. I think network pen testers may find some creative ways to try and detect attacks internally within environment, because that’s what we do. This area has been around for a while. It’s got a number of different names; we call it “hunt teaming”. The name “hunt teaming” was coined by Seth Meisner and Eric Conrad at SANS, and they have done some tremendously amazing, cool work in this arena of actively trying to find the bad guys – not using AV, not trying to use any number of security suites. But assuming that those security tools have failed, how would you detect it?

The reason why we started doing this is because we had a pen test – I think it was early summer – and the customer was completely compromised. We were going through, getting all animated about what we did, and the customer was getting horrified about what we were saying. They said “We must be horrible.” And Joff goes “No, you guys are one of the best customers we’ve ever pen tested, you guys were fantastic!” They about fell over. They were doing everything right and they were still getting compromised. So we were feeling pretty high and mighty, and then they said “We want you guys to come in and find attackers that are using these types of techniques on our network.” And it worked. We found a whole bunch of systems that were compromised in a number of our customers, and it worked out really well.

So I’m going to hand it over to Joff Thyer, and he’s going to go over a couple of slides talking about this. This is kind of a preview of a webcast that we have coming up on Security Weekly on the subject of hunt teaming.

Joff Thyer: Alright, DerbyCon, I love DerbyCon. Thanks John! I’m Joff, I work at Black Hills Information Security. I also co-host Security Weekly, because John doesn’t want to do that anymore – just kidding.

So, hunt teaming – as John was saying, why don’t we flip this model on its head? Let’s find some techniques that we can locate which indicate there’s a compromise in an environment. And I’m not just talking about individual systems, let’s look at the environment network-wide, look at all the systems, look at the behaviors of the systems, look at what’s going on in that environment, look for normal versus abnormal, and see if we can identify abnormal through these various techniques.

So, one example – DNS C2 (see right-hand image). We’ve got stuff we can leverage to look for DNS C2 activity. We can go to our customer and say “Hey guys, why don’t you give me your DNS cache, why don’t you dump that out of your domain controllers or out of your Bind9 environment, whatever it is? I want to look through it. Why don’t you give me logs of your DNS activity over a period of time? We are going to scan through that and we are going to look for interesting things.” One of the first things I did was pull the script together that just scanned through DNS cache, scanned through DNS logs, and just compared it with the DShield’s suspicious domains list. Let’s go see if something pops out. We’ve got to look up to a known bad domain, so let’s flag that and look at that.

Another thing I looked at was whether the records had very short TTL. Turns out that doesn’t have as much attraction as I thought, because when you are looking at DNS cache, things do have short TTLs. They count down, so it’s not quite as effective. But you can sometimes find indicators of the fluxing activity, with NS records in particular, in the short TTL space.

So, what else can we do? We can look at host analysis (see left-hand image). And again, we pick on PowerShell, we assume that we have domain admin access, and we just go to customers and say “Look guys, we just need your administrative access, because we’ve got to run some scripts to hunt around your environment and actually look for interesting things.” So we grab that, we’ve got all kinds of ways we can remote: we can use PsExec, we can use WMIC, or we can use WinRM in some cases. And we can run these scripts on the individual stations and pull back data and look at that data.

So, what are we going to pull back? There’s all sorts of things we want to pull back: we can pull back Run keys, RunOnce key, we can pull back installed software list – we can pull back all kinds of interesting things. So we can pull that back into central location, we can analyze it and we can find some really interesting artifacts.

Also in the host analysis space, we can do HTTP agent strings, for example. Let’s say we have a perimeter of firewall, a proxy – let’s go through all those HTTP agent strings (see right-hand image). What is reaching out and what user agent string are they setting? Let’s take that, let’s sort that, let’s move all the common side stuff aside – we know what Mozilla is, we know what IE is, we know what Safari is. Whatever it might be, look at that and you’ll maybe say “Hey, we got these really low frequency HTTP user agent strings that are really weird. Are these known? Are these supposed to be in our environment? Where are they coming from? What station is using that? Why is it using that HTTP agent string? Let’s chase it down, let’s hunt team it down.” Okay, back to John. That was my five seconds of slides.

John Strand: Like I said, that’s a preview, we don’t have an entire hour on that. Joff had something like 30 slides with all the different scripts, the PowerShell scripts. One of our favorites was dumping the certificates for code signing certificates from your environment and doing an analysis on those. We’ll have an hour of webcast on that coming up.

John Strand: The whole gist of this entire presentation is we need to keep moving forward as network pen testers. We need to continue to find new things to do, not just proving that we can break into organizations, but actually trying to find the clipping levels of where the organizations can detect us. We need to move forward and actually start going through the vulnerability assessment findings and look at the Informationals and the Warnings and the Lows to dig through to find those risks that are missed by just a tool that’s looking for something that’s a High or a Critical.

And the thing that’s hardest for me to say – because it comes across as a bit harsh to other people – right now I believe firmly that the people that are in this room and the pen testing, kind of the offensive security community, are the pointy end of the stick. We are spearheading a lot of what happens in computer security. If you don’t believe me, sit down and talk to anybody that does forensics, or ‘forensicshateor’, as they like to be called from time to time, and ask them “Hey, what do you know about MiniCats?” Many times they’ll just look at you blankly and have no idea. “What do you know about impersonating tokens?” You’ll hear a lot of people in forensics – not all, but you’ll hear a lot of them say “Is that even possible?”

As I said, we are many times the pointy end of the stick. And if we start slacking off, and I see indications of us slacking off, the rest of the industry is going to follow. A lot of the cutting edge research is being done by the people at these Cons, it’s being done by the people at these presentations. And we need to continue to push that as much as we possibly can. So, this is bad for all of us if we allow the things that are occurring to continue to occur. And if you are at a pen testing firm that is a “pen test puppy mill”, as I said, I feel sorry for you. I don’t mean that in kind of a derogatory way, I genuinely feel bad because I come across a lot of people that are just running Nessus results and then turning that into a report and just turning those things as quickly as they possibly can. It’s time to stand up and try to push back a little bit.

I know that that’s hard, I know that that’s difficult, and you are going to fail. And as I say in almost all my presentations, you are going to fail doing the right thing. But I would rather fail doing the right thing than live my entire life doing the compliant thing. That’s not something I want to go to my grave and say “Ah, my environment is PCI compliant.” That’s not what we are here for. That’s not the way we want to go out.

To that end, I created something called “Penetration Testers Code of Ethics”. If you can try to live by this Code of Ethics in what you do in your job and anytime you try to break into anything, you are going to be okay. First, I will never copy and paste automated results, ever. I got into an argument with Ron Gula about that, and he said “You know, I don’t think anybody should ever copy and paste anything from Nessus because that’s our copyrighted work.” I was like “He’s kind of a jerk.” I got to thinking about it, and as time passed I realized he was right, not necessarily because it’s copyright or legal issues, but because it’s lazy. We shouldn’t do that, that’s not what people are paying us to do.

Also, I will never completely trust my scan results. That’s why you go to the Lows and Informationals. That’s where you are going to find beautiful things. That’s where you are going to find vulnerabilities that the tools missed. That’s where you are going to find vulnerabilities that every pen testing firm that came before you missed. And that is awesome. That is a great feeling when someone says “We have hired ten firms before you, and you are the only one to catch this.” That’s probably one of the best feelings in the world in this industry.

I will strive to get caught (after being awesome). Be stealthy, be great at what you do, take over domains, do what you do, but then strive to get caught. And there’s a couple of reasons for this. First reason is we can give them a clipping level. And the second reason is, after you’ve destroyed their network, it gives them a little tip of the hat and you say “Hey, your security team is doing really-really good, they caught us doing these things, but here are some things they can do to make it better.” If you approach pen test as adversarial “they suck, we rock”, you are not going to be very good at this, you’re really not, because your best customers in the future are going to be your current customers. And the more they like you the more you are going to be invited back.

I will go beyond the scan results; I will continue to look at things outside of the organization. We talked about using Recon-ng and finding vulnerabilities that standard scanners don’t pick up. I will try to be a hacker in the original sense of the word. For example, I thought it was pretty cool – I got a Pebble. And I like Pebble, I think it’s a great watch. It’s now completely obsolete and it’s out of date. And then these guys up front – can you guys hold up your arms – show up with those things and I’m like “What the hell are those?” And they’re like “They’re not watches.” I don’t know what that means, but it’s more badass than what I have here.

But that’s cool, right? You go deeper. It isn’t just about getting the T-shirts, it isn’t just about getting the cool packs that we have, it isn’t about getting the cool patches and the stickers and looking the look. Remember being a hacker is a way of life. You take things apart, you’re constantly learning, and most of you are here in this room because you have that already. And we need to continue to have that curiosity about every pen test we do. As I said, you should fall in love with every organization you try to break into, get to know them a little bit. It shouldn’t just be a scan result. Go to their websites. I know that that’s a novel idea for some people, but go to their websites, see what’s there, see what their business is, it’s just kind of a good idea.

I will always stay in scope. This one is actually much harder than it looks. This could be in a presentation all in its own, but I’ll give you one example. We were pen testing a casino, and they gave us their Class C. And we started scanning the Class C, started attacking the Class C. We found a server with the default user ID and password, and we got in, and it was a casino that was completely different than the casino that we were pen testing. And we said “Hey, do you guys own that casino?” And they are like “No…” We said “Well, these are the IP ranges that you gave us,” and they said “Oh yeah, our servers are in that range, we don’t know what they are, we were hoping you could tell us.” Don’t let that happen to you, boys and girls.

And then finally, my reports will rock. I think that’s something that everybody on the team hates me for. But I insist that we have a narrative for every single pen test we did as far as what we did, why we did it, was it successful or was it a failure. A lot of us are insecure about the things that we do that didn’t work. Your customers are paying you for that. It isn’t just about providing a report that says “These are all the successful things that we found, and these are all the vulnerabilities, and look how awesome I am.” If you struggled on something, your customer needs to know that you struggled on something. That’s good. That shows them that there are areas in their network that are fairly strong, and you eventually overcome them hopefully, and then you can explain that to them, but they have to see that struggle. It’s just so important to make sure that it works appropriately.

And now I want to show you kind of a quick preview of Joff’s presentation coming up. We’ve got actual scripts for dumping out, finding out user agent strings, firewall logs, frequency analysis, pivoting activity, how PsExec and all these tools work. We talk about looking at various logs and different PowerShell scripts for all of the different components for looking at workstation variance, going through and pulling out LDAP “computer” objects, going through and pulling out Run and RunOnce execute keys, installed software, looking at the certificate stores – just a lot of cool stuff, so please check that out. And if you want a copy of these slides and you would like to get on the list for the webcast that we have coming up, swing by the Security Weekly booth, bring me your business card or write your email address down, and we will sell it to the vendors. On behalf of all of them, we say thank you. No, we never would do that, ever. That’s not in our DNA.

So, any questions in our last few moments before we get out of here?

Question: What if you don’t block all DLLs, or what if you don’t whitelist all DLLs? And what if you don’t restrict the Run Keys, like RunOnce and RunOnce execute?

John Strand: Those are two different issues. One is persistence, and actually they are both persistence, because a lot of the bad guys, whenever they create backdoors, actually inject themselves into existing Dynamic Link Libraries, and then they persist through that mechanism in many situations. But the issue is what is being written to that directory where DLLs are being executed. A lot of application whitelisting implementations just, basically, start in baby steps, where they list out what directories things are allowed to execute from, and then they slowly start going through and trying to whitelist all the dynamic link libraries from that point on. So, you’ve got to start at that level and slowly work your way up. The idea of trying to whitelist all of them is great, but it’s going to take time to implement, and there are baby steps that you can take for that.

You just run one DLL, but are you whitelisting the JAR files that are being executed as well? What most organizations do is they do “*.jar” – that’s bad, don’t do that. And remember there’s a lot more autostart entry points in Windows than just those, so you have to make sure those are set up as well.

Alright, that’s it. Thank you very much, everybody! I greatly appreciate it! And thanks to Joff for coming up and doing a preview for us.