How to integrate with Log Analytics

This article describes how to enable Log Analytics to monitor & collect logs for cluster pool and cluster operations on HDInsight on AKS. You can enable the integration during cluster pool creation or post the creation. Once the integration at cluster pool is enabled, it isn't possible to disable the integration. However, you can disable the log analytics for individual clusters, which are part of the same pool.

Prerequisites

• Log Analytics workspace. You can think of this workspace as a unique logs environment with its own data repository, data sources, and solutions. Learn how to create a Log Analytics workspace .

  Note

  1. Log Analytics must be enabled at cluster pool level first so as to enable it at a cluster level.

  2. The configuration at cluster pool level is a global switch for all clusters in the cluster pool, therefore all clusters in the same cluster pool can only flow log to one Log Analytics workspace.

Enable Log Analytics using the portal during cluster pool creation

1. Sign in to Azure portal.

2. Select Create a resource and search for cluster pool in marketplace and select Azure HDInsight on AKS Cluster Pool. For more information on starting the cluster pool creation process, see Create a cluster pool.

3. Navigate to the Integrations blade, select Enable Log Analytics.

   

4. From the drop-down list, select an existing Log Analytics workspace. Complete the remaining details required to finish cluster pool creation and select Create.

5. Log Analytics is enabled when the cluster pool is successfully created. All monitoring capabilities can be accessed under your cluster pool’s Monitoring section.

   

Enable Log Analytics using portal after cluster pool creation

1. In the Azure portal search bar, type "HDInsight on AKS cluster pools" and select Azure HDInsight on AKS cluster pools to go to the cluster pools page. On the HDInsight on AKS cluster pools page, select your cluster pool.

   

   

2. Navigate to the "Monitor settings" blade on the left side menu and click on "Configure" to enable Log Analytics.

   

3. Select an existing Log Analytics workspace, and click Ok.

   

Enable Log Analytics using the portal during cluster creation

1. In the Azure portal search bar, type "HDInsight on AKS cluster pools" and select Azure HDInsight on AKS cluster pools to go to the cluster pools page. On the HDInsight on AKS cluster pools page, select the cluster pool in which you want to create a cluster.

   

   

   Note

   It is important to make sure that the selected cluster pool has Log Analytics enabled.

2. Select New Cluster to start the creation process. For more information on starting the cluster creation process, see Create a cluster.

   

3. Navigate to the Integrations blade, select Enable Log Analytics.

4. Select one or more type of logs you would like to collect. Complete the remaining details required to finish the cluster creation and select Create.

   

   Note

   If no option is selected, then only AKS service logs will be available.

5. Log Analytics is enabled when the cluster is successfully created. All monitoring capabilities can be accessed under your cluster's Monitoring section.

   

Enable Log Analytics using portal after cluster creation

1. In the Azure portal top search bar, type "HDInsight on AKS clusters" and select Azure HDInsight on AKS clusters from the drop-down list. On the HDInsight on AKS cluster pools page, select your cluster name from the list page.

   

   

2. Navigate to the "Monitor settings" blade, select Enable Log Analytics. Choose one or more type of logs you would like to collect, and click Save.

   

   Note

   If no option is selected, then only AKS service logs will be available.

Access the log tables and run queries using the portal

1. From the Azure portal, select your cluster pool or cluster of choice to open it.

2. Navigate to the Monitoring section and select the Logs blade to query and analyze the collected data.

   

3. A list of commonly used query templates is provided to choose from to simplify the process or you can write your own query using the provided console.