Solution ideas
This article describes a solution idea. Your cloud architect can use this guidance to help visualize the major components for a typical implementation of this architecture. Use this article as a starting point to design a well-architected solution that aligns with your workload's specific requirements.
This solution idea demonstrates how to use interactive analytics in Azure Data Explorer. It describes how you can examine structured, semi-structured, and unstructured data with improvised, interactive, fast queries.
Jupyter is a trademark of its respective company. No endorsement is implied by the use of this mark. Apache® and Apache Kafka® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.
Architecture
Download a Visio file of this architecture.
Dataflow
- Raw structured, semi-structured, and unstructured (free text) data such as, any type of logs, business events, and user activities can be ingested into Azure Data Explorer from various sources. Ingest the data in streaming or batch mode using various methods.
- Ingest data into Azure Data Explorer with low-latency and high-throughput using its connectors for Azure Data Factory, Azure Event Hubs, Azure IoT Hub, Kafka, and so on. Instead, ingest data through Azure Storage (Blob or ADLS Gen2), which uses Azure Event Grid and triggers the ingestion pipeline to Azure Data Explorer. You can also continuously export data to Azure Storage in compressed, partitioned parquet format and seamlessly query that data as detailed in continuous data export overview.
- Run interactive queries over small to extremely large volumes of data using native Azure Data Explorer tools or alternative tools of your choice. Azure Data Explorer provides many plugins and integrations with the rest of the data platform ecosystem. Use any of the following tools and integrations:
- For interactive analytics, use Azure Data Explorer Web UI, web client for Azure Data Explorer, or Kusto.Explorer, rich windows client for Azure Data Explorer.
- To connect to your Azure Data Explorer cluster, use Jupyter notebooks, Spark connector, any TDS-compliant SQL client, and JDBC and ODBC connections.
- To build new apps or integrate with existing apps or frameworks, use Azure Data Explorer REST APIs and SDKs available in different languages.
- Build near real-time analytics dashboards using Azure Data Explorer dashboards, Power BI, or Grafana.
- Enrich data running federated queries by combining data from SQL database and Azure Cosmos DB using Azure Data Explorer plugins.
Components
- Azure Event Hubs: Fully managed, real-time data ingestion service that's simple, trusted, and scalable.
- Azure IoT Hub: Managed service to enable bi-directional communication between IoT devices and Azure.
- Kafka on HDInsight: Easy, cost-effective, enterprise-grade service for open-source analytics with Apache Kafka.
- Azure Data Factory: Hybrid data integration service that simplifies ETL at scale.
- Azure Data Explorer: Fast, fully managed and highly scalable data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more.
- Azure Data Explorer Dashboards: Natively export Kusto queries that were explored in the Web UI to optimized dashboards.
- Azure Cosmos DB: Fully managed fast NoSQL database service for modern app development with open APIs for any scale.
- Azure SQL DB: Build apps that scale with the pace of your business with managed and intelligent SQL in the cloud.
Scenario details
This solution idea demonstrates how to use interactive analytics with Azure Data Explorer to explore data with improvised, interactive, and fast queries over small to extremely large volumes of data. This data exploration can be done using native Azure Data Explorer tools or alternative tools of your choice. This solution focuses on the integration of Azure Data Explorer with rest of the data platform ecosystem.
Potential use cases
This solution is used by Microsoft customers to track user activity, manage user profiles and user segmentation scenarios.
Contributors
This article is maintained by Microsoft. It was originally written by the following contributors.
Principal author:
- Ornat Spodek | Senior Content Manager
Next steps
For more information, see Azure Data Explorer documentation.