Threat hunting in Microsoft Sentinel

As security analysts and investigators, you want to be proactive about looking for security threats, but your various systems and security appliances generate mountains of data that can be difficult to parse and filter into meaningful events. Microsoft Sentinel has powerful hunting search and query tools to hunt for security threats across your organization's data sources. To help security analysts look proactively for new anomalies that aren't detected by your security apps or even by your scheduled analytics rules, hunting queries guide you into asking the right questions to find issues in the data you already have on your network.

For example, one out of the box query provides data about the most uncommon processes running on your infrastructure. You wouldn't want an alert each time they run. They could be entirely innocent. But you might want to take a look at the query on occasion to see if there's anything unusual.

Hunts in Microsoft Sentinel (preview)

With hunts in Microsoft Sentinel, seek out undetected threats and malicious behaviors by creating a hypothesis, searching through data, validating that hypothesis, and acting when needed. Create new analytic rules, threat intelligence, and incidents based on your findings.

To get started, see Conduct end-to-end proactive threat hunting in Microsoft Sentinel.

Hunting queries

In Microsoft Sentinel, select Hunting > Queries tab to run all your queries, or a selected subset. The Queries tab lists all the hunting queries installed with security solutions from the Content hub, and any extra query you created or modified. Each query provides a description of what it hunts for, and what kind of data it runs on. These queries are grouped by their MITRE ATT&CK tactics. The icons on the right categorize the type of threat, such as initial access, persistence, and exfiltration. MITRE ATT&CK techniques are shown in the Techniques column and describe the specific behavior identified by the hunting query.

Use the queries tab to identify where to start hunting, by looking at result count, spikes, or the change in result count over a 24-hour period. Sort and filter by favorites, data source, MITRE ATT&CK tactic or technique, results, results delta, or results delta percentage. View queries that still need data sources connected, and get recommendations on how to enable these queries.

The following table describes detailed actions available from the hunting dashboard:

Use queries before, during, and after a compromise to take the following actions:

• Before an incident occurs: Waiting on detections isn't enough. Take proactive action by running any threat-hunting queries related to the data you're ingesting into your workspace at least once a week.

  Results from your proactive hunting provide early insight into events that might confirm that a compromise is in process, or at least show weaker areas in your environment that are at risk and need attention.

• During a compromise: Use livestream to run a specific query constantly, presenting results as they come in. Use livestream when you need to actively monitor user events, such as if you need to verify whether a specific compromise is still taking place, to help determine a threat actor's next action, and towards the end of an investigation to confirm that the compromise is indeed over.

• After a compromise: After a compromise or an incident occurred, make sure to improve your coverage and insight to prevent similar incidents in the future.

  • Modify your existing queries or create new ones to assist with early detection, based on insights gained from your compromise or incident.

  • If you discovered or created a hunting query that provides high value insights into possible attacks, create custom detection rules based on that query and surface those insights as alerts to your security incident responders.

    View the query's results, and select New alert rule > Create Microsoft Sentinel alert. Use the Analytics rule wizard to create a new rule based on your query. For more information, see Create custom analytics rules to detect threats.

You can also create hunting and livestream queries over data stored in Azure Data Explorer. For more information, see details of constructing cross-resource queries in the Azure Monitor documentation.

To find more queries and data sources, go to the Content hub in Microsoft Sentinel or refer to community resources like Microsoft Sentinel GitHub repository.

Out of the box hunting queries

Many security solutions include out of the box hunting queries. After you install a solution that includes hunting queries from the Content hub, the out of the box queries for that solution show on the hunting Queries tab. Queries run on data stored in log tables, such as for process creation, DNS events, or other event types.

Many available hunting queries are developed by Microsoft security researchers on a continuous basis. They add new queries to security solutions and fine-tune existing queries to provide you with an entry point to look for new detections and attacks.

Custom hunting queries

Create or edit a query and save it as your own query or share it with users who are in the same tenant. In Microsoft Sentinel, create a custom hunting query from the Hunting > Queries tab.

For more information, see Create custom hunting queries in Microsoft Sentinel.

Livestream sessions

Create interactive sessions that let you test newly created queries as events occur, get notifications from the sessions when a match is found, and launch investigations if necessary. You can quickly create a livestream session using any Log Analytics query.

• Test newly created queries as events occur

  You can test and adjust queries without any conflicts to current rules that are being actively applied to events. After you confirm these new queries work as expected, it's easy to promote them to custom alert rules by selecting an option that elevates the session to an alert.

• Get notified when threats occur

  You can compare threat data feeds to aggregated log data and be notified when a match occurs. Threat data feeds are ongoing streams of data that are related to potential or current threats, so the notification might indicate a potential threat to your organization. Create a livestream session instead of a custom alert rule to be notified of a potential issue without the overheads of maintaining a custom alert rule.

• Launch investigations

  If there's an active investigation that involves an asset such as a host or user, view specific (or any) activity in the log data as it occurs on that asset. Be notified when that activity occurs.

For more information, see Detect threats by using hunting livestream in Microsoft Sentinel.

Bookmarks to keep track of data

Threat hunting typically requires reviewing mountains of log data looking for evidence of malicious behavior. During this process, investigators find events that they want to remember, revisit, and analyze as part of validating potential hypotheses and understanding the full story of a compromise.

During the hunting and investigation process, you might come across query results that look unusual or suspicious. Bookmark these items to refer back to them in the future, such as when creating or enriching an incident for investigation. Events such as potential root causes, indicators of compromise, or other notable events should be raised as a bookmark. If a key event you bookmarked is severe enough to warrant an investigation, escalate it to an incident.

• In your results, mark the checkboxes for any rows you want to preserve, and select Add bookmark. This creates for a record for each marked row, a bookmark, that contains the row results and the query that created the results. You can add your own tags and notes to each bookmark.

  • As with scheduled analytics rules, you can enrich your bookmarks with entity mappings to extract multiple entity types and identifiers, and MITRE ATT&CK mappings to associate particular tactics and techniques.
  • Bookmarks default to use the same entity and MITRE ATT&CK technique mappings as the hunting query that produced the bookmarked results.

• View all the bookmarked findings by clicking on the Bookmarks tab in the main Hunting page. Add tags to bookmarks to classify them for filtering. For example, if you're investigating an attack campaign, you can create a tag for the campaign, apply the tag to any relevant bookmarks, and then filter all the bookmarks based on the campaign.

• Investigate a single bookmarked finding by selecting the bookmark and then clicking Investigate in the details pane to open the investigation experience. View, investigate, and visually communicate your findings by using an interactive entity-graph diagram and timeline. You can also directly select a listed entity to view that entity’s corresponding entity page.

  You can also create an incident from one or more bookmarks, or add one or more bookmarks to an existing incident. Select a checkbox to the left of any bookmarks you want to use, and then select Incident actions > Create new incident or Add to existing incident. Triage and investigate the incident like any other.

• View your bookmarked data directly in the HuntingBookmark table in your Log Analytics workspace. For example:

  

  Viewing bookmarks from the table enables you to filter, summarize, and join bookmarked data with other data sources, making it easy to look for corroborating evidence.

To start using bookmarks, see Keep track of data during hunting with Microsoft Sentinel.

Notebooks to power investigations

When your hunting and investigations become more complex, use Microsoft Sentinel notebooks to enhance your activity with machine learning, visualizations, and data analysis.

Notebooks provide a kind of virtual sandbox, complete with its own kernel, where you can carry out a complete investigation. Your notebook can include the raw data, the code you run on that data, the results, and their visualizations. Save your notebooks so that you can share it with others to reuse in your organization.

Notebooks might be helpful when your hunting or investigation becomes too large to remember easily, view details, or when you need to save queries and results. To help you create and share notebooks, Microsoft Sentinel provides Jupyter Notebooks, an open-source, interactive development, and data manipulation environment, integrated directly in the Microsoft Sentinel Notebooks page.

For more information, see:

• Use Jupyter Notebook to hunt for security threats
• The Jupyter Project documentation
• Jupyter introductory documentation.
• The Infosec Jupyter Book
• Real Python tutorials

The following table describes some methods of using Jupyter notebooks to help your processes in Microsoft Sentinel:

MSTIC, Jupyter, and Python security tools

The Microsoft Threat Intelligence Center (MSTIC) is a team of Microsoft security analysts and engineers who author security detections for several Microsoft platforms and work on threat identification and investigation.

MSTIC built MSTICPy, a library for information security investigations and hunting in Jupyter Notebooks. MSTICPy provides reusable functionality that aims to speed up notebook creation, and make it easier for users to read notebooks in Microsoft Sentinel.

For example, MSTICPy can:

• Query log data from multiple sources.
• Enrich the data with threat intelligence, geolocations, and Azure resource data.
• Extract Indicators of Activity (IoA) from logs, and unpack encoded data.
• Do sophisticated analyses such as anomalous session detection and time series decomposition.
• Visualize data using interactive timelines, process trees, and multi-dimensional Morph Charts.

MSTICPy also includes some time-saving notebook tools, such as widgets that set query time boundaries, select and display items from lists, and configure the notebook environment.

For more information, see:

• MSTICPy documentation
• Jupyter notebooks with Microsoft Sentinel hunting capabilities
• Advanced configurations for Jupyter notebooks and MSTICPy in Microsoft Sentinel

Useful operators and functions

Hunting queries are built in Kusto Query Language (KQL), a powerful query language with IntelliSense language that gives you the power and flexibility you need to take hunting to the next level.

It's the same language used by the queries in your analytics rules and elsewhere in Microsoft Sentinel. For more information, see Query Language Reference.

The following operators are especially helpful in Microsoft Sentinel hunting queries:

• where - Filter a table to the subset of rows that satisfy a predicate.

• summarize - Produce a table that aggregates the content of the input table.

• join - Merge the rows of two tables to form a new table by matching values of the specified columns from each table.

• count - Return the number of records in the input record set.

• top - Return the first N records sorted by the specified columns.

• limit - Return up to the specified number of rows.

• project - Select the columns to include, rename or drop, and insert new computed columns.

• extend - Create calculated columns and append them to the result set.

• makeset - Return a dynamic (JSON) array of the set of distinct values that Expr takes in the group

• find - Find rows that match a predicate across a set of tables.

• adx() - This function performs cross-resource queries of Azure Data Explorer data sources from the Microsoft Sentinel hunting experience and Log Analytics. For more information, see Cross-resource query Azure Data Explorer by using Azure Monitor.

Related articles

• Jupyter notebooks with Microsoft Sentinel hunting capabilities
• Keep track of data during hunting with Microsoft Sentinel
• Detect threats by using hunting livestream in Microsoft Sentinel
• Learn from an example of using custom analytics rules when monitoring Zoom with a custom connector.