The Incident Responder

Matt Presser and his teammates never exactly know what they're walking into when they visit a compromised business - but they know how they must walk out: with the attackers vanquished.

Written by: Dan Kaplan

Matt Presser spent the better part of 2000 like any college junior would have whose study abroad program was in the Mexican resort town of Mazatlán.

He partied and soaked in the sun and waves.

"I sort of considered it my lost semester because not a lot of studying got done," said a laughing Presser, now 42, who was attending New Mexico State University in Las Cruces at the time, about 15 hours away from Mazatlán by car. "Most of my time was spent hanging out and surfing."

But it didn't end up all play for Presser. He decided to get more serious in his second semester abroad, accepting an internship at a family-run computer company.

The job included basic troubleshooting during the early days of the internet. Presser had no previous experience working with PCs – he only passed an introductory to computing class his sophomore year because his friend was the proctor – but as the internship wore on, he started getting good at what he did. He eventually extended his services to private consulting for expats living in Mazatlán who wanted to build web pages or couldn't get their internet to work.

Matt Presser Matt Presser, 42, began his IT security career in academia, eventually landing a job as an assistant professor at New Mexico State University. But eventually his longing for action became too much to bear, and he left to join Trustwave as an incident responder.

Nowadays, as a senior security consultant for Trustwave SpiderLabs, Presser is still helping people figure stuff out in a connected world – except the stakes are much, much higher. He is part of a team of Trustwave experts who help businesses, from Fortune 100 companies to the mom-and-pop shop around the corner, prepare for and resolve security incidents.

The First Cut is the Deepest

Typically, the companies with whom Presser works fall into one of two groups.

The first is a medium-to-large business that Trustwave already has a customer relationship with through its Digital Forensics and Incident Response (DFIR) program. The business has recently experienced an event requiring escalation or forensic investigation, and Trustwave is brought in to help.

In the other camp sit merchants, which can range in size from a single car wash to a major hotel chain, that are believed to have experienced a compromise resulting in real-world fraud.

For the incidents involving existing customers, they usually have some idea that something has gone amiss and can offer background information before Presser or one of his colleagues arrives. Once on scene, he typically starts by scoping the environment to determine what type of data may have been targeted. He then looks for anything strange going on: perhaps a weird network connection or data flowing through unexpected locations.

"Once you've identified some weirdness, then you track down how they got in," Presser said. "You can then start containing and stopping the bleeding." The bandaging process also means helping ensure that wound won't re-open and that the containment is holding.

But ask any incident responder what scares them the most and they'll tell you it's the bleeding they may not be seeing. There is no more sinking feeling than being outsmarted, unable to fully gauge or discover the extent of an incident.

"You might find one bad thing," he said. "But it could be there are 10 other bad things going on."

Sometimes the investigation devolves into a game of whack-a-mole where the incident responder is trying to match wits with the adversary, said Brian Hussey, vice president of cyber threat detection and response who leads the DFIR team at Trustwave. "The attacker may know we are there responding, and they may be trying to change up their tactics and planting more back doors," he said.

The phone rang, and the CISO answered, placing the call on speakerphone. It was the attacker, carrying a thick Eastern European accent. "I just wanted you to know," he said through the phone, "good job, but we will be seeing you again."

In some emergency cases, depending on how quickly a compromise is ballooning, Presser and the team only receive a few hours' notice to hop on a flight. The urgency to control an intensifying breach could also mean working nights, weekends or even holidays. It's not uncommon for attackers to strike during off hours, with the knowledge they may catch their targets off guard and understaffed.

"Hours can be extremely long for an active incident," Presser said "You're working until things are contained. You might get a few hours of sleep here and there."

Presser recently scurried off from his home in Las Cruces – he still resides in the desert city of 100,000 since he arrived there to attend college – to a large U.S. restaurant chain. The company had been hit by the stealthy and insidious Carbanak cybercrime gang, which infests endpoints with difficult-to-flag malware delivered through advanced social engineering methodologies. Ironically, the business had recently read a threat report on Carbanak that Trustwave issued in the winter of 2017 and thought it had shored everything up that would have prevented such an attack from happening there.

It turned out the chain had not battened down its corporate endpoints as well as it should have, and this necessitated a call to Trustwave incident responders to come in and handle the dirty work. Once the outbreak was identified, contained and eradicated, Presser gathered in the CISO's office for a post-mortem discussion.

Something brazen then occurred. The phone rang, and the CISO answered, placing the call on speakerphone. It was the attacker, carrying a thick Eastern European accent. "I just wanted you to know," he said through the phone, "good job, but we will be seeing you again."

Presser couldn't believe his ears. "Holy cow, the gall," he thought.

Shawn Kanady Not only does DFIR Managing Consultant Shawn Kanady play detective and investigator on incident response engagements, he also occasionally assumes the role of quasi-psychologist, counseling victims through the “five stages of grief” following a data breach.

Sure enough, the crooks made good on their word, as well-resourced and determined attackers tend to do. Another wave of malicious phishing attacks soon hit the hotel group. Presser and his comrades returned, again containing the situation within a few days.

Making life easier in this case – and others like it – was that the victim company was an existing customer, having first engaged with Trustwave on an incident response retainer. These agreements typically include incident readiness training, a vital component of a security strategy given how commonplace breaches have become.

Readiness work includes training and tabletop exercises on policy writing, first response, handling client downtime and most importantly of all, identifying incidents. (Businesses that self-detect compromises can typically contain them far quicker than organizations that rely on outside parties to detect compromises, and the 2018 Trustwave Global Security Report showed solid improvement in this area).

Readiness training also helps compromised businesses avoid contaminating the crime scene, which limits the chances of a controlled forensic response.

"Inevitably what ends up happening is they're wrecking evidence because they immediately go into containment mode," said Shawn Kanady, an incident response consultant and Presser's manager. "That information would have helped us determine an initial infection vector."

Speak with a Sales Specialist

Our sales specialists are ready to learn the needs of your business and connect you with the cyber heroes who can help make your security story shine.

Dealing with Enemies – and Emotions

No such concern arises when Presser or Kanady heads to a PCI forensic investigation. In these cases, the victims are in the dark that their walls have been trespassed – that is until someone else tells them first, usually their merchant bank, a card brand or a regulatory body.

Under Payment Card Industry Data Security Standard rules, a merchant must submit to PCI forensic investigation (PFI) if a common point of purchase (CPP) test conducted by issuing banks determines that the merchant served customers who are all now experiencing fraudulent transactions on their credit or debit cards.

Trustwave is one of only a handful of approved PFI investigators and is brought in to instruct the victim business on containment and remediation, as well as prepare a report of findings for card brands and the merchant banks. Depending on the investigator's conclusions, fines can result.

These are often the most complex and tense of responses. Sometimes, Presser said, there are few clues pointing to a compromise because the attackers are long gone. In some cases, investigators wonder if there's been any breach at all. "The threshold for issuing the CPP used to be much higher," Presser said. "Sometimes you just don't have any findings."

Then, there is how the victim reacts. The core competencies of these merchants are typically anything but security and breach response. Bob Russo, the now-retired longtime general manager of the PCI Security Standards Council, used to like to say that the pizza shop down the street knows how to churn out a good slice, but relies on others to help it stay protected. Needless to say, the initial exchanges with these businesses, which usually have unrefined security protocols, are uneasy.

"I'm the shoulder they cry on. I'm the punching bag they hit when they're upset. And I'm also the one that's going to guide through them it."

"These merchants are seeing this [report of a credit card breach] for the first time," Kanady said. "They're out of their minds freaked out. Now you're contacting companies and talking about a topic they're not very comfortable with, something they're forced to deal with. They're worried we're going to audit them or shut them down or fine them. There was one person who thought they were going to go to jail."

The interactions may lead to feelings of resentment toward Trustwave.

"They go through the five stages of grief," Kanady said. "I'm the shoulder they cry on. I'm the punching bag they hit when they're upset. And I'm also the one that's going to guide through them it. It's a real metamorphosis they have to go through."

Sometimes, the breached business' lawyers get involved, only adding to the cacophony of parties wanting answers. And legal involvement leads to another variable: the possibility the victim is not being fully forthright with the investigation.

"Customers could be hiding information because they don't want you to find it," Presser said. "You have to be a detective, but you also have to represent your client. It can be a challenging situation."

Indeed, it's usually at the PFI calls when things can get the strangest with clientele.

"One of the first ones I went one was a large hotel chain," Presser recalled. "We log in and see an anti-virus message that a virus has been detected. One of the kids from IT was with us and says: ‘I've been working here two months and that screen has been like that since I started.' He turns to the CIO and says: ‘I told you about it and nobody's ever done anything.'"

"I just remember the CIO getting really upset, and the kid wasn't allowed to talk to us anymore."

Missing the Chase

Have you had a handy person at your home – perhaps troubleshooting a clogged drain or a shoddy dishwasher – and you find yourself hovering over them as they work, curious if they can find a resolution? It's only human nature that they feel added pressure to not only repair the problem, but also satisfy your personal intrigue.

For an incident responder, that duress is compounded by many factors. But as stressful as situations can get – from tracking down some of the savviest criminal gangs in the world to consoling a vulnerable proprietor to feeling the heat from inquisitive card brands – Presser said he relies on patience, confidence and problem solving to overcome those pressure points.

Presser, a father-of-four When he's not helping businesses prepare for and respond to incidents, Presser, a father-of-four who calls the desert of southeast New Mexico home, is looking for water so he can go wakesurfing. The sport involves a rider trailing behind a motorboat and riding the wake it forms.

In his downtime, he finds solace in wakesurfing with his family – yes, even in the dry terrain of New Mexico, where he's married with four children ranging in age from 5 to 17. It's a hobby he even fits in on some work trips, usually between sessions of training federal law enforcement officials on incident response – one of his other responsibilities.

And it is that draw of helping to police the internet's lawless that shaped Presser's career. Following his return from his college study abroad program, Presser, who had majored in Spanish and economics, decided his future was in IT and information security. He planted his career roots at the college he attended, New Mexico State University, where he took on one of the school's first-ever infosec-specific roles: enterprise systems security administrator. And, working at a college, he earned an early dose of how the cybercriminal mind works.

Colleges are notorious for their decentralization, and the computing paradigm of academia demands far more openness, autonomy and leniency than the corporate world. As an example, professors have been known to stand up rogue web servers that go largely unprotected. This screams dollar signs for the bad guys who use vulnerable networks and endpoints as jumping-off points for their digital assaults against more lucrative targets. "Attackers are generally not going to attack from their own home systems," Presser said. "They route through those various hot points (at colleges and universities) to hide themselves."

While testifying at a Board of Regents hearing in which he provided forensic evidence, Presser impressed New Mexico State officials who asked him if he wanted to teach a computer forensic course. This led to a full-time tenure-track position as an assistant professor teaching information security, but after three semesters, he missed the chase.

"I had a number of really good students and it was definitely rewarding," he said. "I had the beginnings of a popular program, but I missed that hands-on feeling like I was actually catching the bad guys. I missed seeing a return on the work I was putting in."

What hunting down adversaries will look like in the years ahead is anybody's guess, considering how much they've advanced and specialized in just the past decade alone.

In the near term, Kanady said he expects the Trustwave DFIR team to receive an uptick in calls for cryptojacking attacks, a threat that has displaced ransomware among some criminal circles. And if those responsible for these attacks change their motives from stealing CPU resources to mine for digital coins to using the malware as a foothold to infiltrate internal systems and steal sensitive data, the phone will ring start ringing off the hook.

Presser, meanwhile, is ready for whatever the future brings.

"We're here to help," he said.