Create an Event Grid data connection for Azure Data Explorer

1. Browse to your Azure Data Explorer cluster in the Azure portal.

2. Under Data, select Databases > TestDatabase.

   

3. Under Settings, select Data connections and then select Add data connection > Event Grid (Blob storage).

   

4. Fill out the Event Grid data connection form with the following information:

   

   Setting	Suggested value	Field description
   Data connection name	test-grid-connection	The name of the connection that you want to create in Azure Data Explorer. Data connection names can contain only alphanumeric, dash and dot characters, and be up to 40 characters in length.
   Storage account subscription	Your subscription ID	The subscription ID where your storage account is.
   Event type	Blob created or Blob renamed	The type of event that triggers ingestion. Blob renamed is supported only for ADLSv2 storage. To rename a blob, navigate to the blob in Azure portal, right-click on the blob and select Rename. Supported types are: Microsoft.Storage.BlobCreated or Microsoft.Storage.BlobRenamed.
   Storage account	gridteststorage1	The name of the storage account that you created previously.
   Resources creation	Automatic	Turning on automatic resource creation means that Azure Data Explorer creates an Event Grid Subscription, an Event Hubs namespace, and an Event Hubs for you. Otherwise, you need to create these resources manually to ensure the creation of the data connection. See Manually create resources for Event Grid ingestion

   1. Optionally, you can track specific Event Grid subjects. Set the filters for the notifications as follows:

      • Prefix field is the literal prefix of the subject. As the pattern applied is starts with, it can span multiple containers, folders, or blobs. No wildcards are allowed.
        • To define a filter on the blob container, the field must be set as follows: /blobServices/default/containers/[container prefix].
        • To define a filter on a blob prefix (or a folder in Azure Data Lake Gen2), the field must be set as follows: /blobServices/default/containers/[container name]/blobs/[folder/blob prefix].
      • Suffix field is the literal suffix of the blob. No wildcards are allowed.
      • Case-Sensitive field indicates whether the prefix and suffix filters are case-sensitive

      For more information on filtering events, see Blob storage events.

   2. Optionally, you can specify the Data routing settings according to the following information. You don't have to specify all Data routing settings. Partial settings are also accepted.

      Setting	Suggested value	Field description
      Allow routing the data to other databases (Multi database data connection)	Don't allow	Toggle on this option if you want to override the default target database associated with the data connection. For more information about database routing, see Events routing.
      Table name	TestTable	The table you created in TestDatabase.
      Data format	JSON	Supported formats are APACHEAVRO, Avro, CSV, JSON, ORC, PARQUET, PSV, RAW, SCSV, SOHSV, TSV, TSVE, TXT, and W3CLOG. Supported compression options are Zip and Gzip.
      Mapping name	TestTable_mapping	The mapping you created in TestDatabase, which maps incoming data to the column names and data types of TestTable. If not specified, an identity data mapping derived from the table's schema is autogenerated.
      Ignore format errors	Ignore	Toggle on this option if you want to ignore format errors for JSON data format.

      Note

      Table and mapping names are case-sensitive.

   3. Optionally, under Advanced settings, you can specify the Managed identity type that is used by your data connection. By default, System-assigned is selected.

      If you select User-assigned, you need to manually assign a managed identity. If you select a user that isn't assigned to your cluster yet, it will be auto-assigned. For more information, see Configure managed identities for your Azure Data Explorer cluster.

      If you select None, the storage account and Event Hub are authenticated via connection strings. This method isn't recommended.

      

5. Select Create

1. Browse to your storage account in the Azure portal.
2. Select Events.
3. In the Get Started tab, select the Azure Data Explorer tab.
4. In the Create continuous data ingestion tile, select Create.

Choose destination to route events into

1. Select a cluster in which you want to ingest data.
2. Select a target database.
3. Select Select.

Create data connection

1. Fill out the form with the following information:

   

   Setting	Suggested value	Field description
   Data connection name	test-grid-connection	The name of the connection that you want to create in Azure Data Explorer.
   Storage account subscription	Your subscription ID	The subscription ID where your storage account is.
   Event type	Blob created or Blob renamed	The type of event that triggers ingestion. Blob renamed is supported only for ADLSv2 storage. To rename a blob, navigate to the blob in Azure portal, right-click on the blob and select Rename. Supported types are: Microsoft.Storage.BlobCreated or Microsoft.Storage.BlobRenamed.
   Storage account	gridteststorage1	The storage account from which you accessed this wizard is autopopulated.
   Resources creation	Automatic	Turning on automatic resource creation means that Azure Data Explorer creates an Event Grid Subscription, an Event Hubs namespace, and an Event Hubs for you. Otherwise, you need to create these resources manually to ensure the creation of the data connection. See Manually create resources for Event Grid ingestion

   1. Optionally, you can track specific Event Grid subjects. Set the filters for the notifications as follows:

      • Prefix field is the literal prefix of the subject. As the pattern applied is starts with, it can span multiple containers, folders, or blobs. No wildcards are allowed.
        • To define a filter on the blob container, the field must be set as follows: /blobServices/default/containers/[container prefix].
        • To define a filter on a blob prefix (or a folder in Azure Data Lake Gen2), the field must be set as follows: /blobServices/default/containers/[container name]/blobs/[folder/blob prefix].
      • Suffix field is the literal suffix of the blob. No wildcards are allowed.
      • Case-Sensitive field indicates whether the prefix and suffix filters are case-sensitive

      For more information on filtering events, see Blob storage events.

   2. Optionally, you can specify the Data routing settings according to the following information. You don't have to specify all Data routing settings. Partial settings are also accepted.

      Setting	Suggested value	Field description
      Allow routing the data to other databases (Multi database data connection)	Don't allow	Toggle on this option if you want to override the default target database associated with the data connection. For more information about database routing, see Events routing.
      Table name	TestTable	The table you created in TestDatabase.
      Data format	JSON	Supported formats are APACHEAVRO, Avro, CSV, JSON, ORC, PARQUET, PSV, RAW, SCSV, SOHSV, TSV, TSVE, TXT, and W3CLOG. Supported compression options are Zip and Gzip.
      Mapping name	TestTable_mapping	The mapping you created in TestDatabase, which maps incoming data to the column names and data types of TestTable. If not specified, an identity data mapping derived from the table's schema is autogenerated.
      Ignore format errors	Ignore	Toggle on this option if you want to ignore format errors for JSON data format.

      Note

      Table and mapping names are case-sensitive.

   3. Optionally, under Advanced settings, you can specify the Managed identity type that is used by your data connection. By default, System-assigned is selected.

      If you select User-assigned, you need to manually assign a managed identity. If you select a user that isn't assigned to your cluster yet, it will be auto-assigned. For more information, see Configure managed identities for your Azure Data Explorer cluster.

      If you select None, the storage account and Event Hub are authenticated via connection strings. This method isn't recommended.

      

2. Select Create.

The following example shows an Azure Resource Manager template for adding an Event Grid data connection. You can edit and deploy the template in the Azure portal by using the form.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "namespaces_eventhubns_name": {
            "type": "string",
            "defaultValue": "eventhubns",
            "metadata": {
                "description": "Specifies the event hub namespace name."
            }
        },
        "EventHubs_eventhubdemo_name": {
            "type": "string",
            "defaultValue": "eventhubdemo",
            "metadata": {
                "description": "Specifies the event hub name."
            }
        },
        "consumergroup_default_name": {
            "type": "string",
            "defaultValue": "$Default",
            "metadata": {
                "description": "Specifies the consumer group of the event hub."
            }
        },
        "StorageAccounts_storagedemo_name": {
            "type": "string",
            "defaultValue": "storagedemo",
            "metadata": {
                "description": "Specifies the storage account name"
            }
        },
        "Clusters_kustocluster_name": {
            "type": "string",
            "defaultValue": "kustocluster",
            "metadata": {
                "description": "Specifies the name of the cluster"
            }
        },
        "databases_kustodb_name": {
            "type": "string",
            "defaultValue": "kustodb",
            "metadata": {
                "description": "Specifies the name of the database"
            }
        },
        "tables_kustotable_name": {
            "type": "string",
            "defaultValue": "kustotable",
            "metadata": {
                "description": "Specifies the name of the table"
            }
        },
        "mapping_kustomapping_name": {
            "type": "string",
            "defaultValue": "kustomapping",
            "metadata": {
                "description": "Specifies the name of the mapping rule"
            }
        },
        "dataformat_type": {
            "type": "string",
            "defaultValue": "csv",
            "metadata": {
                "description": "Specifies the data format"
            }
        },
             "databaseRouting_type": {
            "type": "string",
            "defaultValue": "Single",
            "metadata": {
                "description": "The database routing for the connection. If you set the value to **Single**, the data connection will be routed to a single database in the cluster as specified in the *databaseName* setting. If you set the value to **Multi**, you can override the default target database using the *Database* EventData property."
            }
        },
        "dataconnections_kustodc_name": {
            "type": "string",
            "defaultValue": "kustodc",
            "metadata": {
                "description": "Name of the data connection to create"
            }
        },
        "subscriptionId": {
            "type": "string",
            "defaultValue": "[subscription().subscriptionId]",
            "metadata": {
                "description": "Specifies the subscriptionId of the resources"
            }
        },
        "resourceGroup": {
            "type": "string",
            "defaultValue": "[resourceGroup().name]",
            "metadata": {
                "description": "Specifies the resourceGroup of the resources"
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "[resourceGroup().location]",
            "metadata": {
                "description": "Location for all resources."
            }
        }
    },
    "variables": {
    },
    "resources": [{
            "type": "Microsoft.Kusto/Clusters/Databases/DataConnections",
            "apiVersion": "2022-02-01",
            "name": "[concat(parameters('Clusters_kustocluster_name'), '/', parameters('databases_kustodb_name'), '/', parameters('dataconnections_kustodc_name'))]",
            "location": "[parameters('location')]",
            "kind": "EventGrid",
            "properties": {
                "managedIdentityResourceId": "[resourceId('Microsoft.Kusto/clusters', parameters('clusters_kustocluster_name'))]",
                "storageAccountResourceId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroup'), 'Microsoft.Storage/storageAccounts', parameters('StorageAccounts_storagedemo_name'))]",
                "eventHubResourceId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroup'), 'Microsoft.EventHub/namespaces/eventhubs', parameters('namespaces_eventhubns_name'), parameters('EventHubs_eventhubdemo_name'))]",
                "consumerGroup": "[parameters('consumergroup_default_name')]",
                "tableName": "[parameters('tables_kustotable_name')]",
                "mappingRuleName": "[parameters('mapping_kustomapping_name')]",
                "dataFormat": "[parameters('dataformat_type')]",
                "databaseRouting": "[parameters('databaseRouting_type')]"
            }
        }
    ]
}

The following code sample uses the Azure Blob Storage SDK to upload a file to Azure Blob Storage. The upload triggers the Event Grid data connection, which ingests the data into Azure Data Explorer.

var azureStorageAccountConnectionString = <storage_account_connection_string>;
var containerName = <container_name>;
var blobName = <blob_name>;
var localFileName = <file_to_upload>;
var uncompressedSizeInBytes = <uncompressed_size_in_bytes>;
var mapping = <mapping_reference>;
// Create a new container if it not already exists.
var azureStorageAccount = new BlobServiceClient(azureStorageAccountConnectionString);
var container = azureStorageAccount.GetBlobContainerClient(containerName);
container.CreateIfNotExists();
// Define blob metadata and uploading options.
IDictionary<String, String> metadata = new Dictionary<string, string>();
metadata.Add("rawSizeBytes", uncompressedSizeInBytes);
metadata.Add("kustoIngestionMappingReference", mapping);
var uploadOptions = new BlobUploadOptions
{
    Metadata = metadata,
};
// Upload the file.
var blob = container.GetBlobClient(blobName);
blob.Upload(localFileName, uploadOptions);

The following code sample uses the Azure Data Lake SDK to upload a file to Data Lake Storage Gen2. The upload triggers the Event Grid data connection, which ingests the data into Azure Data Explorer.

var accountName = <storage_account_name>;
var accountKey = <storage_account_key>;
var fileSystemName = <file_system_name>;
var fileName = <file_name>;
var localFileName = <file_to_upload>;
var uncompressedSizeInBytes = <uncompressed_size_in_bytes>;
var mapping = <mapping_reference>;
var sharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey);
var dfsUri = "https://" + accountName + ".dfs.core.windows.net";
var dataLakeServiceClient = new DataLakeServiceClient(new Uri(dfsUri), sharedKeyCredential);
// Create the filesystem.
var dataLakeFileSystemClient = dataLakeServiceClient.CreateFileSystem(fileSystemName).Value;
// Define file metadata and uploading options.
IDictionary<String, String> metadata = new Dictionary<string, string>();
metadata.Add("rawSizeBytes", uncompressedSizeInBytes);
metadata.Add("kustoIngestionMappingReference", mapping);
var uploadOptions = new DataLakeFileUploadOptions
{
    Metadata = metadata,
    Close = true // Note: The close option triggers the event being processed by the data connection.
};
// Upload the file.
var dataLakeFileClient = dataLakeFileSystemClient.GetFileClient(fileName);
dataLakeFileClient.Upload(localFileName, uploadOptions);

Rename blobs

In ADLSv2, it's possible to rename directories. However, it's important to note that renaming a directory doesn't trigger blob renamed events or initiate the ingestion of blobs contained within the directory. If you want to ensure the ingestion of blobs after renaming a directory, you should directly rename the individual blobs within the directory.

The following code sample shows how to rename a blob in an ADLSv2 storage account.

var accountName = <storage_account_name>;
var accountKey = <storage_account_key>;
var fileSystemName = <file_system_name>;
var sourceFilePath = <source_file_path>;
var destinationFilePath = <destination_file_path>;
var sharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey);
var dfsUri = "https://" + accountName + ".dfs.core.windows.net";
var dataLakeServiceClient = new DataLakeServiceClient(new Uri(dfsUri), sharedKeyCredential);
// Get a client to the the filesystem

var dataLakeFileSystemClient = dataLakeServiceClient.GetFileSystemClient(fileSystemName);
// Rename a file in the file system
var dataLakeFileClient = dataLakeFileSystemClient.GetFileClient(sourceFilePath);
dataLakeFileClient.Rename(destinationFilePath);