When I first came to White Oak Security, I was presented with a unique opportunity to do some training for threat hunting assessments. Prior to that point, I had some background knowledge of threat hunting, but never any formal training. After taking some online courses (shout-out to Chris Sanders’ Threat Hunting course, it’s an excellent introduction to the topic), I was able to apprentice on some threat hunting assessments under a more experienced member of our team.
I wanted to share some insights into the topic of threat hunting from a penetration testing point of view to help other pentesters understand what it is and how training in threat hunting can improve a pentester’s approach to their role.
What Is Threat Hunting?
I’ll give a brief description for those who aren’t familiar with the concept of threat hunting. Threat hunters examine logs and other artifacts in computer systems or network devices to try and identify malicious activity. Threat hunting is typically conducted using raw AV/EDR logs or record aggregating systems such as SIEM applications.
Threat hunting requires knowledge of how user or system actions are reflected in logs and other records, as well as what patterns of these records may indicate. For example, let’s say we’re looking at log records for web activity. If we notice a large number of records from a single system conducting POST requests to a login endpoint, it could be indicative of a brute-force attempt by a malicious actor to gain access to the system via password guessing or credential stuffing.
Threat hunting requires knowledge of these types of patterns, as well as the ability to determine or analyze unknown patterns discovered during the hunting process. To be a skilled Threat Hunter, one must understand what differentiates benign activity from malicious action.
Threat Hunting Tools
Typically, a threat hunter will need the following resources to conduct a successful hunt:
- Event sources, log data, or event records of some type and access to aggregation systems
- A host for developing tools or scripts that can access SIEM devices on the network
- Information about the environment in which the hunt is being conducted, including systems and network structure
- Internal team contacts, including incident response and product owners
- A clearly defined plan including which activities to hunt for and what expected deliverables will be
Threat Hunting Process
A simplified explanation of a threat hunting process is as follows… Initially, we determine a particular and specific area of focus for the search. Some example topics might be:
- Malicious authentication issues
- Bad actor persistence in a network
- Specific types of malware
- Unauthorized software used by employees which could pose a security threat
- Generalized evidence of increased malicious activity which could indicate a trend
It’s often best to narrow down exactly the activity you want to look for, as it’s very easy to become sidetracked due to the nebulous nature of such data.
We then want to answer these important questions:
- How would this activity be manifested?
- If you were performing these actions, what sorts of log records would be generated?
- What Windows Event codes would be logged?
- In which order would the actions be performed?
- What specific characteristics of the behavior would identify it in log records?
- Are there specific patterns that make it stand out from normal traffic?
It’s then essential to determine a baseline of behavior; i.e. determine what is normal for the environment. If you’re looking at HTTP network traffic, determine which behavior happens routinely and is identifiable as benign and expected. Identify which systems are the “big talkers” and which ones are rare to see. This will help better eliminate false positives from the search and make malicious behavior stand out.
We’ll then start examining anomalies. Does that “quiet talker” host start making large numbers of requests that seem out of place? Why? Are there hosts making requests which don’t make sense? Look into why this is happening. It’s useful to research the teams managing the systems and services in question to understand why this behavior is manifesting and to be able to eliminate similar strange, but benign, activities later on. Being able to separate out anomalies from the noise is definitely an art and not a science. But it gets easier with time and more exposure to the environment.
When anomalies are detected, it’s important to then examine them in detail and determine why they are occurring. This usually requires partnership with product owners or other interested parties, as well as IR (Incident Response) teams, to identify whether the behavior is benign or malicious and, if so, halt the behavior and secure the system in question.
Threat Hunting Vs Penetration Testing
Because I wrote this blog addressing pentesters, I’ll write from that perspective. You may wonder why I’m detailing this process and the reason is that I believe a knowledge of threat hunting tactics can assist penetration testers and red teamers in doing their jobs.
As pentesters/red teamers, we frequently seek to evade detection to emulate threats. The knowledge of how activities we perform manifest themselves in logs is a crucial piece of information when performing these types of assessments. If we want to emulate a specific threat actor by performing certain actions, we should know how these actions will be represented and how the IR team should detect them. Not only from a standpoint of being able to provide good recommendations for future improvements in detection, but also to identify gaps in detection that could be exploited in later, more advanced activity. For example, does the IR team have a detection rule built around large numbers of failed Windows Event 4768 or just 4769?
In addition to guiding activities, while emulating specific threats, general knowledge of threat hunting can greatly improve assessments where the goal is to remain undetected as long as possible. When planning a new action during the assessment, it’s essential to understand what logs will be generated in SIEM systems, what signatures might be tripped in detection systems, and what patterns may reveal the intent behind our behaviors. Knowing what methods may be used to detect activity can allow us to consider alternative patterns of behavior, such as adding jitter or time spacing to actions, using alternative and unexpected paths to execute code, or mimicking the behavior of other benign systems.
I don’t suggest every pentester or red teamer become a threat hunter, but taking a deeper look at your opponent and how they conduct their tactics is always a valuable exercise and increases the value for your clients. It allows you to understand ways that detection teams could identify you on an assessment and helps to improve your own personal processes and techniques.
White Oak Security’s Approach To Threat Hunting
At White Oak Security, the process for threat hunting with a client starts with collaboratively choosing topics. We discuss with the client the threats they are concerned about, which ones they currently have detection for, and which ones they don’t. After identifying potential hunting opportunities, we work together to prioritize that list and determine where best to begin. Typically, there are 2-3 main topics per hunt unless a particular topic might be more time intensive, with a typical hunt taking roughly a week to complete.
Once hunt topics are decided upon, White Oak Security gathers information from the client regarding available data sources which may cover any log evidence related to the hunting topic. Many hunts are focused on a single SIEM system aggregating log sources, although such evidence might span across multiple aggregators.
Additionally, White Oak Security and the client team decide whether hunting should be performed only manually or if scripting tools to reproduce the hunt in an automated fashion should be developed during the project also. Then the hunting begins!
After being granted access to the client’s SIEM aggregation tools, White Oak Security’s threat hunting team begins to formulate queries to identify threat scenarios within the log records. We do this by conducting detailed research into threat actor TTPs (Tactics, Techniques, and Procedures) and determining how these behaviors would manifest in log artifacts. Additionally, the team determines new and novel ways that these behaviors might have been performed in order to catch unanticipated malicious events. The next step is to craft queries, formatted for the specific SIEM tools in use, to create proof of concept searches for the malicious behavior in question.
After initial queries are created, the data in the SIEM system is carefully analyzed. White Oak Security works closely with the client team and reports events that seem to align with the malicious behavior being hunted. Once the client team has examined these events, if deemed potentially malicious, they are then passed on to the client’s IR team for immediate action.
If desired by the client team and agreed upon at the start of the project, successful queries are then selected by the White Oak Security team for inclusion into scripted tools. The White Oak Security team writes code to connect to the SIEM system in question, run the crafted hunt queries against the current data, and output reports identifying potentially malicious behavior. These reports can then be reviewed by the client and integrated into other systems. The scripted tools can then be delivered to the client for future hunting.
At the end of the project, White Oak Security provides the client with a playbook of the queries developed during the project and a corresponding explanation of how to conduct this sort of hunt on their own systems in the future. These playbooks give the client teams the ability to repeat these hunts internally when desired.
Threat Hunter Conclusion
Threat hunting is a valuable activity for any organization, regardless of size. Knowing what activity is being performed in your environment and proactively looking for threats meets bad actors head-on and decreases the amount of time before they are discovered. However, detection is only as good as available sources of data. It’s also important to know how reliable your data sources are; if they don’t actually contain the data needed to hunt or the data is unreliable, it makes detecting malicious actors that much more difficult. Even the best defensive detection mechanisms have gaps and threat hunting is the way to patch those gaps – by identifying where detection does not currently have coverage (or where that coverage isn’t working) and providing improved methods for future detection.
As we’ve discussed, threat hunting knowledge can also better inform pentesters on the impact of their tactics and techniques, as well as assist them with improving the detections of their Blue Team partners.
Threat hunting malicious behavior in seas of log data sometimes seems like finding a needle in a haystack. White Oak Security’s goal is to create better magnets to discover those needles efficiently and effectively.
In a future blog, we’ll discuss more specific scenarios of problems encountered during threat hunting, how we resolved them, and give some examples of anonymized case studies.
MORE FROM WHITE OAK SECURITY
White Oak Security is a highly skilled and knowledgeable cyber security and penetration testing company that works hard to get into the minds of opponents to help protect those we serve from malicious threats through expertise, integrity, and passion.
