Threat hunting: how organisations can go looking for new attacks

Detection and response solutions work well to a point, but many in cyber security believe organisations need to be more proactive. We speak to Talos security lead Martin Lee about the rising practice of threat hunting and whether it's something organisations should be pursuing.

In the past, IT security practices have really revolved around detection and response. Security teams monitor enterprise networks and investigate automated alerts that are generated from a range of tooling such as Security Information and Event Management (SIEM) systems, using their expertise to identify, escalate, and employ mitigation strategies where needed. Of course, the enterprise security market is vast, with thousands of vendors all providing differing tools and services, although at their core, many of these depend on core principles of using a working knowledge of the security threat landscape to respond to issues as they occur.

Increasingly, though, attacks are becoming more complicated and threat actors are becoming more evasive. There is no doubt that a well-reasoned approach to tooling and a mature security profile and culture still prove the most important mitigators of most attacks, however, some actors can still squeeze through the perimeter and cause real damage. In a recent Nominet survey of 300 CISOs, only 34% noted that they were "somewhat or slightly confident" in their organisation's choice of security solutions, while only 17% indicated that the "array of technology making up their stack" was completely effective.

That has led many to start investigating a more proactive approach, and one of the latest trends in proactive cyber security approaches is threat hunting. Threat hunting essentially describes the process of actively scouring enterprise networks and data looking for evidence of abnormal or illicit behaviour left behind by attackers. As Gartner describes, organisations that employ threat hunting use an analyst-centric approach to uncover hidden threats that slip by automated prevention and detection controls. It is a very human-centric approach, requiring a team of highly skilled IT security professionals (threat hunters), who have an in-depth knowledge of systems, security, data analysis and possess a distinct sense of creativity in their work.

Threat hunting is generally about trying to uncover threats before they cause damage or raise alerts. Thus, it is an investigative effort, with researchers following their nose based on their previous experience and expertise. Discoveries made by threat hunters can lead to serious damage mitigation, with their newly created techniques often being turned into automated, rule-based algorithms designed to weed out further instances of the threats they uncover. However, threat hunting can be quite difficult for organisations to employ and it certainly isn't for everyone.

To find out more about this increasingly popular practice, we spoke to Martin Lee, security lead at Talos, who has been involved with the security industry for 16 years. Lee leads a team of threat hunters tasked with identifying the most significant, novel threats that have potential to cause organisations real damage. Talos holds a relatively interesting position within the threat hunting sphere, as the organisation does not directly operate under a commercial model. Rather, Talos supports the Cisco security portfolio with a threat hunting capability that allows the networking giant to stay abreast of the varying methods that threat actors are coming up with.

In partnering with Cisco, Talos can mine and search the vast amount of telemetry that the networking giant has available to identify what is different or significant, allowing them to actively discover new threats. We spoke to Lee about how his team actually goes on the hunt for new threats and what factors to consider when assessing whether an organisation should be considering adding a threat hunting capability as part of their security postures. He says that threat hunting, for the more experienced cyber researchers, is really about identifying ‘where the sport is', with his team working to uncover only the most highly sophisticated attacks that modern day tooling can't take down on its own.

How does modern day threat hunting actually work and what does the process involve?

For me, it's all about enabling threat hunters to do their job. An increasing trend in the realm of security - and by necessity threat hunting - is around automating processes that support the actual people that are doing the hunting themselves. 

I think security teams sometimes have the tendency to get bogged down in spending a lot of time in what is almost working a production line, carrying out some very tightly defined routine tasks to look for things. Increasingly, automated processes are handling these aspects of the process, taking away that repetitive work and freeing up people to do what they do best, which is to provide their skills, professional experience, knowledge, and sense of innovation and creativity to come up with the new techniques of identifying the latest threats. 

There is always a risk - in applying tightly defined, tried and tested processes to find the bad guys - that you'll only identify older techniques, which not what we're looking for. If we're to identify the latest techniques, then we need to be creative, we need a lot of innovation, and we need people to think about what threat actors might be doing now and the traces they might leave behind in data. Using this mentality, we can come up with the techniques to identify those traces.

A big part of this is adopting a process of rapid innovation and experimentation, spending a fair amount of time trying out a range of different new techniques. You also have to prepare for the reality that most of the new techniques that hunters come up with are not going to work but failing fast is key. You can't be putting a lot of effort into one technique that is clearly not promising. We keep experimenting until we uncover a new technique, we then dig down into how that technique works, refining and analysing the technique, and eventually turn it into an automated process that would then throw up alerts for a researcher to look at in detail. 

The key thing here is having automated systems that are supporting the threat hunters in their work. What we want to do is enable people to be threat hunters in the sense that they use their experience to 'follow their nose', giving them a capacity to research, uncover and identify problems. 

So, is it really about combining our understandings of the existing threat landscapes with an educated conceptualisation of how threat actors might look to expand into new territories and techniques? 

I'd say it's about understanding that threat hunting is still a very novel environment. IT, and the way we use technology, is changing very rapidly and the way that threat actors are abusing technology is changing even faster. I think we must recognise that we don't actually have the answers. What we can do is build on our previous knowledge and techniques, while innovating faster. 

We need to recognise that there is more out there that we need to uncover. We need new ways of analysing data and new ways of thinking about the landscape, so we can secure the networks and systems that we rely on.

When exactly should an organisation start to consider employing threat hunting?

It requires a certain amount of maturity in security posture. Anything within security - and indeed within business - is all about return on investment. I would say that many organisations would get the best ROI in doing the basics correctly. Organisations need to have first accomplished things like applying endpoint protection on every single system, employing a mature approach to patching, and instilling robust approach to user and device authentication, before they consider something like threat hunting. 

As part of developing this profile, those systems are creating a nice amount of security-focused data that can be analysed and mined. So, once organisations get the basics right to the point that a CISO or executive is happy with its basic security posture, it then becomes beneficial to have a threat hunting team in the environment. They can analyse the data that you're generating to understand what's wrong, what's different, and what might threat actors look like within the network. 

The other thing that you need is the ability to do something with that information. Certainly, if you have a threat hunting team, you need to have the incident and response team sitting alongside them, so when the threat hunters find something the IR team can respond to it in good time. As I say, it all comes down to ROI and thinking about where resources might be best invested. I would strongly suspect that, for most organisations, that a threat hunting team wouldn't be first on a firm's list of where to spend their dollars. 

In that sense, is it something that only larger enterprise organisations would benefit from? Is threat hunting not really something for an SME to explore? 

That discussion should be based around risk and security posture. There are SMBs that have an exceptionally good security posture and they equally have an awful lot of risk. They may well benefit from having a threat hunting capability in-house. Certainly, larger organisations with a mature security posture tend to be the ones that have an in-house threat intelligence team, but it's not essentially a 'large-firm only' type of thing.

There is also a sense of indirectly benefiting from a security vendor's employment of a threat hunting team, as when you use the the products and services of that vendor, you're taking advantage of a threat hunting capability as well. In that regard, you don't necessarily have to employ your own threat hunting capability in order to benefit from it.

Talking about the modes of employing threat hunting, is this something that organisations should merely outsource, or is there value in building out in-house threat hunting teams? Are there a series of factors that firms should consider? 

If you think about the concept of having layers of security that sit on top of each other, you can apply this equally to threat hunting and threat intelligence. As I just talked about, you're always going to benefit from third-party threat intelligence and threat hunting through the security products that you have in place. So that's one layer of threat hunting, but then organisations may think about some kind of some kind of outsourced TR or TI function on top of that. 

To be honest though, personally, I think you're always going to get the best level of service and the best understanding of your business if you have (threat hunting) as an in-house function. That's not to say that you won't benefit from third-party services, but if you develop an in-house threat hunting system that actually understands your own network, and what makes it different from anyone else, you'll be putting yourself in the best possible position when something unusual happens. 

Attackers generally will leave their sticky fingerprints at the scene of the crime and there is always something that you can pick up on. The more familiar you are with those specific systems and networks that you're defending, the more likely you are to be able to identify when something isn't quite right. 

Again, it does all come down to ROI. But if you're in a good position with solid security posture - and your risk analysis team indicates that spending money on an in-house TH team would be beneficial - that's when it would make sense. 

Could you talk about an example of a case where Talos was able to hunt out and mitigate a threat that other security teams weren't aware of?

The best example that I can think of was the work we did around VPNFilter, which was an attack that was put together by a nation state threat actor. They had compromised 500,000 predominantly small/home office routers with a very sophisticated, modular piece of malware. In this case, there was clearly a large attack in preparation.

We were able to identify this attack before it happened. We identified that there were routers behaving incorrectly, making connections that were not commanded. We then looked at why, finding out about the malware infecting these systems and how it was communicating with its controller through the command and control infrastructure. 

Then, we alerted the right people, using our relationship with Cisco as well as our partnerships with public sector bodies to work with law enforcement. This allowed us to get the command and control infrastructure that the attack used taken down, so that the attacker was no longer in the position to launch the attack that they had planned.

In regard to the threat intelligence teams themselves, what kinds of resources and data do they need access to, in order to effectively undertake the activity?

The two most important things within the threat hunting team are actually the people that you have and the culture in which they work. People tend to focus on the idea that it's all about data, or about having a scene that gives you visibility of that data. Personally, I tend to disagree. 

There is more than enough data available for threat hunting to be effective and that data doesn't have to be internal. There are a lot of external sources out there that you can use to mine, query, and ask questions of, but the key thing is having the right people in place. Threat hunting requires people with an innate sense of curiosity, with a drive to find out more about the world and what's going on. You certainly need some experience and a background or at least a knowledge of security. Most importantly, though, having people with that sense of curiosity and having an environment where that curiosity is encouraged, is fundamental. 

That said, you do need some data and a layer of tooling that allows you to adequately question the data. One pitfall that you can go down is actually having tooling that is too strong, presenting the data only in one way. That puts your hunting team on rails, meaning you can only ever go in one direction and you can only ask questions that your toolset will give you answers to. 

This approach is far too rigid, as we are still learning a lot about the discipline. What you want to do is enable this experimental, innovative, and creative environment where hunters can play with the data and come up with new questions to ask the data. That is a far better method of identifying when things are going wrong.