Manage clusters

Clusters are individual compute workloads such as Apache Spark, Apache Flink, and Trino, which can be created rapidly in few minutes with preset configurations and few clicks.

This article describes how to manage a cluster using Azure portal.

Note

You are required to have an operational cluster, Learn how to create a cluster.

Get started

  1. Sign in to Azure portal.

  2. In the Azure portal search bar, type "HDInsight on AKS clusters" and select "Azure HDInsight on AKS clusters" from the drop-down list.

    Screenshot showing search option for getting started with HDInsight on AKS Cluster.

  3. Select your cluster name from the list page.

    Screenshot showing selecting the HDInsight on AKS Cluster you require from the list.

View cluster details

You can view the cluster details in the "Overview" blade of your cluster. It provides general information and easy access to the tools that are part of the cluster.

Property Description
Resource group The resource group in which cluster is created.
Cluster pool name Cluster pool name inside which the cluster is created.
Cluster type The type of the cluster such as Spark, Trino, or Flink.
HDInsight on AKS version HDInsight on AKS cluster version. For more information, see versioning.
Cluster endpoint The endpoint of the cluster.
Cluster package Component versions associated with the cluster.
Subscription details Subscription name and subscription ID.
Location Region in which the cluster is deployed.
Cluster size details Node size, node type, and number of nodes.

Screenshot showing all cluster settings overview on HDInsight on AKS Cluster.

Manage cluster size

You can check and modify the number of worker nodes for your cluster using "Cluster size" blade in the Azure portal. There are two options to scale up/down your cluster:

Screenshot showing cluster size blade on HDInsight on AKS Cluster.

Manage cluster access

HDInsight on AKS provides a comprehensive and fine-grained access control at both control plane and data plane, which allows you to manage cluster resources and provide access to cluster data plane.

Learn how to manage access to your cluster.

Configure secure shell (SSH)

Secure shell (SSH) allows you to submit jobs and queries to your cluster directly. You can enable or disable SSH using "Secure shell (SSH)" blade in the Azure portal.

Note

Enabling SSH will create additional VMs in the cluster. The maximum allowed secure shell nodes are 5.

Screenshot showing SSH blade on HDInsight on AKS Cluster.

Manage cluster configuration

HDInsight on AKS allows you to tweak the configuration properties to improve performance of your cluster with certain settings. For example, usage or memory settings. In the Azure portal, use "Configuration management" blade of your cluster to manage the configurations.

You can do the following actions:

  • Update the existing service configurations or add new configurations.
  • Export the service configurations using RestAPI.

Learn how to manage the cluster configuration.

View service details

In the Azure portal, use "Services" blade in your cluster to check the health of the services running in your cluster. It includes the collection of the services and the status of each service running in the cluster. You can drill down on each service to check instance level details.

Learn how to check service health.

Enable integration with Azure services

In the Azure portal, use "Integrations" blade in your cluster pool to configure the supported Azure services. Currently, we support Log Analytics and Azure managed Prometheus and Grafana, which has to be configured at cluster pool before you can enable at cluster level.

Screenshot showing cluster integration with log analytics on HDInsight on AKS Cluster.

Delete cluster

Deleting a cluster doesn't delete the default storage account nor any linked storage accounts. You can re-create the cluster by using the same storage accounts and the same metastores.

From the "Overview" blade in the Azure portal:

  1. Select Delete from the top menu.

    Screenshot showing how to delete the cluster on HDInsight on AKS Cluster.

  2. Status can be checked on Notification icon Screenshot showing the Notifications icon in the Azure portal..