The SANS Institute has introduced a course to train security personnel to detect and remediate Advance Persistent Threats (APT), sophisticated and surreptitious attacks, generally to conduct industrial/commercial/government espionage. Security Bistro spoke with security, incident response and forensics expert, Rob Lee, instructor for the course, “Advanced Computer Forensic Analysis and Incident Response,” about understanding and combating APT. In the first two posts, Lee spoke about the nature and sources of the APT threat. In this post, Lee speaks about the skills and experience needed to detect and remediate these threats, the goals of the course and the unending nature of the battle against APT.
Security Bistro: Let’s talk about the “P” in APT. What do we mean by “persistent” and what are the implications for targeted organizations?
Rob Lee: It goes back to Incident Response 101. For many years, if you discovered a system that was compromised, you’d pull the plug: limit the data exposure, stop the hemorrhaging. That’s the worst thing you can do with advanced threats. It tells adversary they’ve been detected, and as soon as they think they’ve been detected, they might think a full-scale remediation is ongoing and they will spread even further inside the network. It essentially becomes a game of whack a mole. You discover one system, take it down and another system will pop up. Then the system you remediated before starts beaconing out again, then a third. Most organizations freak out a little when we tell them this: “Wait a minute, you’re telling me that once we discover where the threats are on the network, not to do anything about it?”
No, that’s not what we are saying. Forensicate the machine, pull out as much threat intelligence as possible, so you know exactly what this adversary looks like on the network. Use this information to scope out the entire enterprise, identifying every single node that is compromised. The you have to simultaneously take all the compromised machines offline at once, and that will essentially takes APT off your network —today. Then you implement additional security measures.
SB: I have a feeling you’re going to tell me that isn’t the end of it, though.
Lee: Usually, once they realize they have lost contact totally with the target, they will start another campaign, usually the next day. Additional spear-phishing emails will come in trying to re-compromise the target. That’s what makes it persistent. They won’t give up trying to gain access to your network; in many cases, they will be successful again, because they will employ different techniques. They start to use Technique 2.0. They will use the least capable version of their tools that will work. They’ll start with a baseball bat; you pull out a knife, they pull out a gun. You pull out a machine gun, they pull out a bazooka. This escalation will eventually stop when you force the adversary to go into R&D mode again. It’s a longer window to deploy a new capability to counter your techniques. They need to come up with a new magic trick. But you will slow them down long enough to give yourself a little breathing room.
SB: So, there is no end?
Lee: We tell this to executives: Once you are victim of APT you will be dealing with it forever. You are not going to go back to a time you were pre-APT. This is the kind of thing you will be dealing with from a security perspective from this day forward. It’s a matter of how many waves occurred this month.
SB: How does an organization have a hint that it may have been compromised and where do they need to look.
Lee: There are two ways. One, the FBI or someone calls you and says, “We can’t tell why, but you might want to go look at this IP address; we’re pretty sure that you’ve been compromised. 90% of the organizations find out they’ve been compromised by an APT because a third party informs them, which is kind of frustrating because you’re blind up to that point.
The second way is that you are in some sort of sharing group, like FS-ISAC [Financial Services Information Sharing and Analysis center] for the financial industry. Other industries have their own organizations. At these meeting, you’ll be told so and so got compromised last week and here is the current profile the attackers are using. You might want to check out your network. So they are learning on the fly what the adversaries are doing.
The best analogy is that essentially your security is like the TSA. And it’s as effective as the TSA too until they know exactly what to look for. You can’t search every grandma in the line and expect to find a terrorist. But it’s new that the terrorists are potentially using underwear bombs; you’ve received some sort of intelligence and you start looking for this specific type of threat. You can’t just look blindly at every piece of questionable item going on in your network. It’s too intense. The one thing that is missing in a lot of security organizations is a threat intelligence matrix that is honing data and directing their team to look for that type of stuff going on in their network. This information is not highly publicized. to a certain extent, most organizations treat it as classified.
SB: What types of investigatory skills are required?
Lee: There are two types of individuals: One is a host-based forensicator, someone who can look within a system and quickly ascertain whether that system is probably compromised.
The second individual, the network forensicator, is the one who can do deep packet inspection off interesting traffic that they have identified. The one piece that is missing is if you are organization that has these types of individuals but has 500,000 hosts. How do I even tell them what to go look at? This is where what we call threat intelligence comes into play. Typically, an organization will have some sense of what it might be looking for. This is kind of hard, especially for organizations that are new to this. But once an organization has been hit, they’re able to start building a profile of the types of connections that are going on, the types of footprints they should be looking for.
SB: How does this APT course help?
Lee: Most organizations simply need individuals that have the skills needed to detect unknown APT on their network. Basically comes down to this. The FBI, for example, won’t give you any information on what to look for. They’re just going to give you an IP address. And that organization is going to need individuals who are going to be able to go through that system and identify where an APT is, where did they go, what did they take and can we now use that information to start building threat intelligence to be able to scope out the rest of our network. You identify the system that was compromised. You have to fully forensicate it, identify the malware, have the reverse engineering team start working on the malware to determine where the aggressors went, forensicate those additional systems and start to identify what types of information they were after, because your executives need to know what was taken. They know they were breached, but they want to know, how bad is it?
Most organizations imply don’t have personnel trained to be able to answer these hard questions. So my goal in the training class is to create a new type of instant responder; one who is able to investigate advanced threats and begin in counter them using the techniques that we highlight in the class.
SB: How do you simulate real-life scenarios?
The key to why the training is so effective is I’ve spent a year setting up a real Windows enterprise network. We made it as realistic as possible, based on the Department of Defense Host-Based Security System. We had the network online for almost a year prior to comprising it. We also had personnel play the play roles of employees, so there was normal looking data on the network. We also did full patching to make the network as realistic as possible. And then we compromised the network just like the APT would: spear-phishing, attacks, data movement, data collection, data exfiltration — all the APT techniques and tools.
In the end what we have here — I’m kind of a proud papa — is one of most realistic case scenarios to get people exposed to that hard-learned experience without going through a situation in which you were dealing with APT in your own organization.
SB: What is your ultimate goal?
Lee: 90% of the organizations out there have not been able to detect APT. We are not going to be “winning” the battle until 50% of the organizations can self-detect their own compromises. This class will help organizations become part of that 50%.
Rob Lee is an entrepreneur and consultant in the Washington DC area, specializing in information security, incident response, and digital forensics. Rob is currently the curriculum lead and author for digital forensic and incident response training at the SANS Institute in addition to owning his own firm. Rob has more than 15 years of experience in computer forensics, vulnerability and exploit discovery, intrusion detection/prevention and incident response. Rob was also a director for Mandiant, a company focused on investigating advanced adversaries, such as the APT, for four years prior to starting his own business.