Migrate from Splunk to Azure Monitor Logs

Article
03/20/2024

Azure Monitor Logs is a cloud-based managed monitoring and observability service that provides many advantages in terms of cost management, scalability, flexibility, integration, and low maintenance overhead. The service is designed to handle large amounts of data and scale easily to meet the needs of organizations of all sizes.

Azure Monitor Logs collects data from a wide variety of sources, including Windows Event logs, Syslog, and custom logs, to provide a unified view of all Azure and non-Azure resources. Using a sophisticated query language and curated visualization you can quickly analyze millions of records to identify, understand, and respond to critical patterns in your monitoring data.

This article explains how to migrate your Splunk Observability deployment to Azure Monitor Logs for logging and log data analysis.

For information on migrating your Security Information and Event Management (SIEM) deployment from Splunk Enterprise Security to Azure Sentinel, see Plan your migration to Microsoft Sentinel.

Why migrate to Azure Monitor?

The benefits of migrating to Azure Monitor include:

Fully managed, Software as a Service (SaaS) platform with:
- Automatic upgrades and scaling.
- Simple per-GB pay-as-you-go pricing.
- Cost optimization and monitoring features and low-cost Basic logs.
Cloud-native monitoring and observability, including:
Native integration with a range of complementary Azure services, such as Microsoft Sentinel for security information and event management, Azure Logic Apps for automation, Azure Managed Grafana for dashboarding, and Azure Machine Learning for advanced analysis and response capabilities.

Compare offerings

Splunk offering	Product	Azure offering
Splunk Platform	Splunk Cloud Platform Splunk Enterprise	Azure Monitor Logs is a centralized software as a service (SaaS) platform for collecting, analyzing, and acting on telemetry data generated by Azure and non-Azure resources and applications.
Splunk Observability	Splunk Infrastructure Monitoring Splunk Application Performance Monitoring Splunk IT Service Intelligence	Azure Monitor is an end-to-end solution for collecting, analyzing, and acting on telemetry from your cloud, multicloud, and on-premises environments, built over a powerful data ingestion pipeline that's shared with Microsoft Sentinel. Azure Monitor offers enterprises a comprehensive solution for monitoring cloud, hybrid, and on-premises environments, with network isolation, resilience features and protection from data center failures, reporting, and alerts and response capabilities. Azure Monitor's built-in features include: Azure Monitor Insights - ready-to-use, curated monitoring experiences with pre-configured data inputs, searches, alerts, and visualizations. Application Insights - provides Application Performance Management (APM) for live web applications. Azure Monitor AIOps and built-in machine learning capabilities - provide insights and help you troubleshoot issues and automate data-driven tasks, such as predicting capacity usage and autoscaling, identifying and analyzing application performance issues, and detecting anomalous behaviors in virtual machines, containers, and other resources. These features are free of installation fees.
Splunk Security	Splunk Enterprise Security Splunk Mission Control Splunk SOAR	Microsoft Sentinel is a cloud-native solution that runs over the Azure Monitor platform to provide intelligent security analytics and threat intelligence across the enterprise.

Introduction to key concepts

Azure Monitor Logs	Similar Splunk concept	Description
Log Analytics workspace	Namespace	A Log Analytics workspace is an environment in which you can collect log data from all Azure and non-Azure monitored resources. The data in the workspace is available for querying and analysis, Azure Monitor features, and other Azure services. Similar to a Splunk namespace, you can manage access to the data and artifacts, such as alerts and workbooks, in your Log Analytics workspace. Design your Log Analytics workspace architecture based on your needs - for example, split billing, regional data storage requirements, and resilience considerations.
Table management	Indexing	Azure Monitor Logs ingests log data into tables in a managed Azure Data Explorer database. During ingestion, the service automatically indexes and timestamps the data, which means you can store various types of data and access the data quickly using Kusto Query Language (KQL) queries. Use table properties to manage the table schema, data retention and archive, and whether to store the data for occasional auditing and troubleshooting or for ongoing analysis and use by features and services. For a comparison of Splunk and Azure Data Explorer data handling and querying concepts, see Splunk to Kusto Query Language map.
Basic and Analytics log data plans		Azure Monitor Logs offers two log data plans that let you reduce log ingestion and retention costs and take advantage of Azure Monitor's advanced features and analytics capabilities based on your needs. The Analytics plan makes log data available for interactive queries and use by features and services. The Basic log data plan provides a low-cost way to ingest and retain logs for troubleshooting, debugging, auditing, and compliance.
Archiving and quick access to archived data	Data bucket states (hot, warm, cold, thawed), archiving, Dynamic Data Active Archive (DDAA)	The cost-effective archive option keeps your logs in your Log Analytics workspace and lets you access archived log data immediately, when you need it. Archive configuration changes are effective immediately because data isn't physically transferred to external storage. You can restore archived data or run a search job to make a specific time range of archived data available for real-time analysis.
Access control	Role-based user access, permissions	Define which people and resources can read, write, and perform operations on specific resources using Azure role-based access control (RBAC). A user with access to a resource has access to the resource's logs. Azure facilitates data security and access management with features such as built-in roles, custom roles, inheritance of role permission, and audit history. You can also configure workspace-level access and table-level access for granular access control to specific data types.
Data transformations	Transforms, field extractions	Transformations let you filter or modify incoming data before it's sent to a Log Analytics workspace. Use transformations to remove sensitive data, enrich data in your Log Analytics workspace, perform calculations, and filter out data you don't need to reduce data costs.
Data collection rules	Data inputs, data pipeline	Define which data to collect, how to transform that data, and where to send the data.
Kusto Query Language (KQL)	Splunk Search Processing Language (SPL)	Azure Monitor Logs uses a large subset of KQL that's suitable for simple log queries but also includes advanced functionality such as aggregations, joins, and smart analytics. Use the Splunk to Kusto Query Language map to translate your Splunk SPL knowledge to KQL. You can also learn KQL with tutorials and KQL training modules.
Log Analytics	Splunk Web, Search app, Pivot tool	A tool in the Azure portal for editing and running log queries in Azure Monitor Logs. Log Analytics also provides a rich set of tools for exploring and visualizing data without using KQL.
Cost optimization		Azure Monitor provides tools and best practices to help you understand, monitor, and optimize your costs based on your needs.

1. Understand your current usage

Your current usage in Splunk will help you decide which pricing tier to select in Azure Monitor and estimate your future costs:

Follow Splunk guidance to view your usage report.
Azure Monitor cost estimates using the Pricing Calculator.

2. Set up a Log Analytics workspace

Your Log Analytics workspace is where you collect log data from all of your monitored resources. You can retain data in a Log Analytics workspace for up to seven years. Low-cost data archiving within the workspace lets you access archived data quickly and easily when you need it, without the overhead of managing an external data store.

We recommend collecting all of your log data in a single Log Analytics workspace for ease of management. If you're considering using multiple workspaces, see Design a Log Analytics workspace architecture.

To set up a Log Analytics workspace for data collection:

Create a Log Analytics workspace.

Azure Monitor Logs creates Azure tables in your workspace automatically based on Azure services you use and data collection settings you define for Azure resources.
Configure your Log Analytics workspace, including:
1. Pricing tier.
2. Link your Log Analytics workspace to a dedicated cluster to take advantage of advanced capabilities, if you're eligible, based on pricing tier.
3. Daily cap.
4. Data retention.
5. Network isolation.
6. Access control.
Use table-level configuration settings to:
1. Define each table's log data plan.
  
  The default log data plan is Analytics, which lets you take advantage of Azure Monitor's rich monitoring and analytics capabilities.
2. Set a data retention and archiving policy for specific tables, if you need them to be different from the workspace-level data retention and archiving policy.
3. Modify the table schema based on your data model.

3. Migrate Splunk artifacts to Azure Monitor

To migrate most Splunk artifacts, you need to translate Splunk Processing Language (SPL) to Kusto Query Language (KQL). For more information, see the Splunk to Kusto Query Language map and Get started with log queries in Azure Monitor.

This table lists Splunk artifacts and links to guidance for setting up the equivalent artifacts in Azure Monitor:

Splunk artifact	Azure Monitor artifact
Alerts	Alert rules
Alert actions	Action groups
Infrastructure Monitoring	Azure Monitor Insights are a set of ready-to-use, curated monitoring experiences with pre-configured data inputs, searches, alerts, and visualizations to get you started analyzing data quickly and effectively.
Dashboards	Workbooks
Lookups	Azure Monitor provides various ways to enrich data, including: - Data collection rules, which let you send data from multiple sources to a Log Analytics workspace, and perform calculations and transformations before ingesting the data. - KQL operators, such as the join operator, which combines data from different tables, and the externaldata operator, which returns data from external storage. - Integration with services, such as Azure Machine Learning or Azure Event Hubs, to apply advanced machine learning and stream in additional data.
Namespaces	You can grant or limit permission to artifacts in Azure Monitor based on access control you define on your Log Analytics workspace or Azure resource groups.
Permissions	Access management
Reports	Azure Monitor offers a range of options for analyzing, visualizing, and sharing data, including: - Integration with Grafana - Insights - Workbooks - Dashboards - Integration with Power BI - Integration with Excel
Searches	Queries
Source types	Define your data model in your Log Analytics workspace. Use ingestion-time transformations to filter, format, or modify incoming data.
Data collections methods	See Collect data for Azure Monitor tools designed for specific resources.

For information on migrating Splunk SIEM artifacts, including detection rules and SOAR automation, see Plan your migration to Microsoft Sentinel.

4. Collect data

Azure Monitor provides tools for collecting data from log data sources on Azure and non-Azure resources in your environment.

To collect data from a resource:

Set up the relevant data collection tool based on the table below.
Decide which data you need to collect from the resource.
Use transformations to remove sensitive data, enrich data or perform calculations, and filter out data you don't need, to reduce costs.

This table lists the tools Azure Monitor provides for collecting data from various resource types.

Resource type	Data collection tool	Similar Splunk tool	Collected data
Azure	Diagnostic settings		Azure tenant - Microsoft Entra audit logs provide sign-in activity history and audit trail of changes made within a tenant. Azure resources - Logs and performance counters. Azure subscription - Service health records along with records on any configuration changes made to the resources in your Azure subscription.
Application	Application insights	Splunk Application Performance Monitoring	Application performance monitoring data.
Container	Container insights	Container Monitoring	Container performance data.
Operating system	Azure Monitor Agent	Universal Forwarder, Heavy Forwarder	Monitoring data from the guest operating system of Azure and non-Azure virtual machines.
Non-Azure source	Logs Ingestion API	HTTP Event Collector (HEC)	File-based logs and any data you send to a data collection endpoint on a monitored resource.

5. Transition to Azure Monitor Logs

A common approach is to transition to Azure Monitor Logs gradually, while maintaining historical data in Splunk. During this period, you can:

Use the Log ingestion API to ingest data from Splunk.
Use Log Analytics workspace data export to export data out of Azure Monitor.

To export your historical data from Splunk:

Use one of the Splunk export methods to export data in CSV format.
To collect the exported data:
1. Use Azure Monitor Agent to collect the data you export from Splunk, as described in Collect text logs with Azure Monitor Agent.
  
  or
2. Collect the exported data directly with the Logs Ingestion API, as described in Send data to Azure Monitor Logs by using a REST API.

Next steps

Learn more about using Log Analytics and the Log Analytics Query API.
Enable Microsoft Sentinel on your Log Analytics workspace.
Take the Analyze logs in Azure Monitor with KQL training module.

Share via