The Threat Hunters

A group of super-skilled security pros are using breakthrough technologies to uncover and respond to cyberattacks - and doing it from far, far away.

Written by: Dan Kaplan

Aaron Wooten was just 6 years old in 1991 when his father brought home the family’s first computer. Fully aware of his son’s interest in electronics, he tasked Wooten with figuring out how to use the now-nostalgic hunk of machinery.

Instead of first powering on the PC and hammering away at the keyboard like your average first-grader living three decades ago might, Wooten immediately guided his curiosity to the motherboard.


“I opened up the internals, and I pulled out the BIOS chip,” he recalled. “Back then, you could pull it out. Except when I attempted to put it back in, something exploded. I had five hours to repair it myself before my dad got home.”

The youngster embarked on a frantic trial-and-error operation, successfully navigating around the risks of a child toying with complex hardware (“The risk of electric shock was actually quite high,” Wooten coyly admits now) to successfully troubleshoot the malfunction. He avoided any bodily harm, his father never knew what happened and his raison d'être was permanently etched.

When you look at a typical IT department…they have people doing security things, but they’re not threat hunting.

James Antonakos

“Since then, I’ve had an interest in how things work,” said Wooten, now a managing consultant at Trustwave and part of the company’s elite SpiderLabs team of researchers, testers, forensic investigators and incident responders.

Wooten, like other computer prodigies – many of whom make up the ranks at Trustwave – has parlayed a penchant for cyber into a lifelong pursuit. While some may have fallen to the dark side, choosing to take up a career in criminal hacking, Wooten and his peers have steered clear of the unsavory world of digital crime to impart their craftsmanship on helping others. Most recently he is accomplishing this through a relatively new IT security practice known as threat hunting.

Except Wooten and his cohorts are trading binoculars and camouflage for state-of-the-art tools and methods that allow them to dig deep inside an organization’s network to stealthily collect data, search for anomalous activity and determine if a data breach has taken place.

Threat hunting wasn’t always feasible – or desired for that matter. Sure, there has been some level of “hunting” going on for years, especially as data-loss incidents, and now ransomware attacks, have taken center stage within the business risk arena.

What You Might Discover in a Threat Hunt:

  • Ports open to the outside world (RDP, TeamViewer, and others)
  • Evidence of suspicious events in log files (DNS, web, firewall)
  • Ineffective endpoint protection
  • Unusual website access
  • Machine communication at odd hours or data being exfiltrated
  • Common networking tools harvesting credentials

Yet responders were limited in what they could learn when reacting to a suspected attack for two primary reasons: 1) Organizations would eventually draw a line in the sand when manual, on-site investigations became too costly and they became willing to accept any additional risk and 2) technology hadn’t advanced far enough for investigations to be conducted remotely with marked success.

But organizations didn’t seem to care either. “People weren’t thinking about what happens if things go wrong,” recalled Mike Wilkinson, a Trustwave SpiderLabs managing consultant who prior to his work here spent seven years in law enforcement investigating crimes that involved computers. “Historically you have that mindset that it won’t happen to us.”

But the tide is turning. With 2017 on pace to set yet another record number of breaches, data security requirements stiffening around the globe, and outside firms like Trustwave leveraging custom-developed and advanced threat intelligence – businesses are waking up to their newfound ability to partner with a provider that can both respond and investigate in the event of a suspected incident, as well as proactively identify and contain suspicious activity before it can inflict significant harm.

And that’s a big-time revelation for companies, which continue to lack the capabilities themselves to identify the indicators and causes of sophisticated attacks and impede the exfiltration of sensitive data. This fact is congruous with the well-documented security skills shortage facing the industry, where an estimated one million security jobs go unfilled worldwide.

“Most organizations hire minimal IT staff to keep their network running,” said James Antonakos, a Trustwave SpiderLabs incident response consultant and a longtime computer science professor. “They’re not malware analysts. They’re not forensics people. They may not really understand the whole attack chain and where you look for things and how you stop them. They have their hands full.”

Antonakos got his start as an electrical engineer before becoming a computer scientist and security researcher. He credits his background with not only molding him to become a proficient security professional but also opening his eyes to the importance of programming scripts and developing tools that solve problems. Likening threat hunting to signals processing in electronics, Antonakos thinks of electric currents as no different than traffic and packets traversing a network.

“If you take the rate of occurrence between connections form a particular system on your network and plot that over time for every endpoint, you get lots of graphs within one big overall graph,” he said. “Patterns show up. It’s very hard for a human (inside a business) to look in to that.”

Especially at your average organization, where security maturity remains a choppy work in progress.

“When you look at a typical IT department, they have syslog servers gathering all their logs, SIEMs that events feed into, and they write some rules and get alerts generated,” Antonakos said. “That’s a very reactive way of doing security. But that’s what organizations do because they don’t have threat hunters there. They have people doing security things, but they’re not threat hunting.”

“With threat hunting, you are using information you’re getting, but you’re allowing a human to decide what to look for next or react to what they’re seeing,” he added. “Researchers are now putting artificial intelligence in the extrapolation of data from databases that contain all of the events that happened in the organization…This allows you to see those really drawn-out, slow attacks that a human wouldn’t be able to correlate.”

For Wooten, the biggest draw of threat hunting is its ability to rummage for cybercrime evidence on a much larger scale. And he knows a thing or two about being restricted by distance, growing up in Perth, Australia, which is considered one of the most isolated cities in the world.

Now, instead of analyzing a select number of systems that would need to be manually studied in person, he and his team can use automated means to gain visibility across an entire network – even one that stretches globally – in real time. The resulting scalability is what wows him.

“We can deploy our agents and search for IOCs (indicators of compromise) across networks of hundreds of thousands of PCs,” he said.

And it is this ability that further advances the argument that organizations, especially the ones that already have strong security in place, should adopt proactive threat hunting. Instead of waiting for the inevitable breach, which could lead to a disastrous egress of confidential assets, the functionality now exists for regular check-ups that can verify that the security technologies that organizations have in place are doing their job and that the bad guys don’t have a foothold across the network. It can also point out something more general, like poor security hygiene and a failure to map where data lives.

For example, Wilkinson described one case in which his team identified IP addresses within a network that were behaving strangely. The threat hunters turned that information over to the customer, which took three weeks to physically identify the offending machines – they were stored away, apparently unknowingly, in a cabinet somewhere. In another hunt, Wilkinson turned up a “Pokemon Go” mining operation in which a member of the IT team was using several systems to catch the animated creatures.

“Sometimes we learn things about the customer that they weren’t even aware of,” he said.

Proactive threat hunting will never yield a guaranteed bill of health, but it goes a long way toward mitigating risk. “It’s unlikely we’re going to say your systems are completely clean,” Wooten said. “It’s not a statement we like to make. You don’t know what you don’t know. But what we can do is give you a degree of assurance.”

Aaron Wooten maintains his childhood curiosity as he works to help organizations discover threats and respond to malicious activity.

One particularly vulnerable component of the IT environment are endpoints, which can range from desktops and laptops to smartphones or Internet of Things devices. Hackers like to start small and go after soft targets, which brings the added benefit of not raising suspicion or exerting too many resources. Endpoints fit that bill well. They are considered the most vulnerable part of the network and are often operated by users who are more than willing to lend a helping hand to attackers, predominantly through email phishing scams and website-based “watering hole” ambushes.

Antonakos and his fellow SpiderLabs threat hunters utilize several tools that reveal shenanigans happening on the endpoint. How a tool typically works: Trustwave sends a small software sensor to an organization’s IT person. They install it on their systems, and Trustwave threat hunters then manage alerts and information being generated by those sensors. “We have really good visibility into what is happening on the endpoint and control over it,” he said. “Our tools tie in with threat intelligence feeds that will alert on malicious files and block them from executing.”

When their day is over, and the Trustwave investigators return from the mercurial networks that are so often their hunting grounds, they are never deterred from the fight.

“The reason that we do what we do is to help people and make a difference,” said Wooten, whose desire for occupational nobility predated his work at Trustwave, when he spent three years developing the security architecture for the Australian government’s child abuse royal commission. “Without that sense of satisfaction, I don’t think many people would be in this role. It’s a high-pressure environment, but the reward is you are making a difference.”

For Wilkinson, who fell in love with security when he (innocuously) executed a buffer overflow attack on a popular anti-virus product in 1997 while bored in a college class, the gratification and bliss also comes from not knowing what is behind every door.

“No network is identical to another network,” he said. “We’re always coming across a program we haven’t seen before, a different configuration we haven’t seen before, different security products. It’s a never-ending learning experience, which is part of the attraction. It keeps you young.”

Maybe they can’t stay 6 years old forever, extracting BIOS chips while Dad is gone for a few hours, but that doesn’t mean the fun needs to end for these threat hunters. Turns out, little Aaron Wooten was on to something.

Lightning Round

Three Questions for the Hunters

James Antonakos

  • Why did you decide to pursue a career in security?
  • My interest in computer networking led to further interest in computer and network security. I began consulting part-time in information security 16 years ago, and after a great 32-year teaching career, I decided to teach online part-time and do security work full-time. This grew from a desire to help more people and more companies become secure, beyond the security graduates from my two-year degree program.
  • What are the key characteristics of becoming a master threat hunter and incident responder?
  • Every hunt and every IR engagement is different, so it is important to keep a fresh and open mind about how you approach the work. You also need to understand the inner workings of operating systems, all details of computer networking, and read, read, read to stay current with malware trends.
  • What do you hope to change about the way organizations think about security?
  • That by the findings of a threat hunt or IR engagement the organization clearly sees the gaps in its security and takes seriously the steps I’ve recommended to become more secure. In addition, that the organization learns to promote a “culture of security,” where every employee understands they play a role in keeping the organization secure.

Aaron Wooten

  • Why did you decide to pursue a career in security?
  • I’ve always had a natural curiosity about what makes things tick. So, the technical aspect of the job, combined with my desire to help people, made security an easy choice.
  • What are the key characteristics of becoming a master threat hunter and incident responder?
    1. Determination: There will be times during an investigation where the evidence will be very thin. It’s important to stick to your guns and take your time. Most attackers end up making a mistake at some point.
    2. Objectivity: Approaching an investigation with an open mind is very important to avoid tunnel vision. As an investigator, you must simply present the facts and let the evidence speak for itself.
    3. Compassion: A key part of providing this sort of service is communicating with stakeholders during a crisis. You need to be extra understanding and reassuring to people who may be under a great deal of pressure.
  • What do you hope to change about the way organizations think about security?
  • I think that as organizations are now more exposed to the impact of large scale data breaches, the mindset is changing from “if my company is breached” to “when my company is breached.” I’d like to see this continue. I feel that while the general public has become much more accepting of breaches occurring, they are also demanding a lot more transparency about data protection and the incident management process.

Mike Wilkinson

  • Why did you decide to pursue a career in security?
  • I am a glutton for punishment! In all seriousness, I enjoy the challenge of constant learning and problem solving.
  • What are the key characteristics of becoming a master threat hunter and incident responder?
  • Persistence, lateral thinking and the ability to think like an attacker. There is also what I call the “that’s strange” factor, which is the ability to look at a set of data and notice things that are out of place.
  • What do you hope to change about the way organizations think about security?
  • In an ideal world, organizations should not have to think about security. Unfortunately, we do not live in an ideal world. Security should be part of business-as-usual, a consideration of every business process.