Manually scale Azure HDInsight clusters

HDInsight provides elasticity with options to scale up and scale down the number of worker nodes in your clusters. This elasticity allows you to shrink a cluster after hours or on weekends. And expand it during peak business demands.

Scale up your cluster before periodic batch processing so the cluster has adequate resources. After processing completes, and usage goes down, scale down the HDInsight cluster to fewer worker nodes.

You can scale a cluster manually using one of the following methods. You can also use autoscale options to automatically scale up and down in response to certain metrics.

Utilities to scale clusters

Microsoft provides the following utilities to scale clusters:

Using any of these methods, you can scale your HDInsight cluster up or down within minutes.

Impact of scaling operations

When you add nodes to your running HDInsight cluster (scale up), jobs remains unaffected. New jobs can be safely submitted while the scaling process is running. If the scaling operation fails, the failure leaves your cluster in a functional state.

If you remove nodes (scale down), pending or running jobs fail when the scaling operation completes. This failure is because of some of the services restarting during the scaling process. Your cluster may get stuck in safe mode during a manual scaling operation.

The impact of changing the number of data nodes varies for each type of cluster supported by HDInsight:

• Apache Hadoop

  You can seamlessly increase the number of worker nodes in a running Hadoop cluster without impacting any jobs. New jobs can also be submitted while the operation is in progress. Failures in a scaling operation are gracefully handled. The cluster is always left in a functional state.

  When a Hadoop cluster is scaled down with fewer data nodes, some services are restarted. This behavior causes all running and pending jobs to fail at the completion of the scaling operation. You can, however, resubmit the jobs once the operation is complete.

• Apache HBase

  You can seamlessly add or remove nodes to your HBase cluster while it's running. Regional Servers are automatically balanced within a few minutes of completing the scaling operation. However, you can manually balance the regional servers. Log in to the cluster headnode and run the following commands:

  pushd %HBASE_HOME%\bin
  hbase shell
  balancer
  

  For more information on using the HBase shell, see Get started with an Apache HBase example in HDInsight.

  Note

  Not appliable for Kafka clusters.

• Apache Hive LLAP

  After scaling to N worker nodes, HDInsight automatically set the following configurations and restart Hive.

  • Maximum Total Concurrent Queries: hive.server2.tez.sessions.per.default.queue = min(N, 32)
  • Number of nodes used by Hive's LLAP: num_llap_nodes = N
  • Number of Node(s) for running Hive LLAP daemon: num_llap_nodes_for_llap_daemons = N

How to safely scale down a cluster

Scale down a cluster with running jobs

To avoid having your running jobs fail during a scale down operation, you can try three things:

1. Wait for the jobs to complete before scaling down your cluster.
2. Manually end the jobs.
3. Resubmit the jobs after the scaling operation has concluded.

To see a list of pending and running jobs, you can use the YARN Resource Manager UI, following these steps:

1. From the Azure portal, select your cluster. The cluster is opened in a new portal page.

2. From the main view, navigate to Cluster dashboards > Ambari home. Enter your cluster credentials.

3. From the Ambari UI, select YARN on the list of services on the left-hand menu.

4. From the YARN page, select Quick Links and hover over the active head node, then select Resource Manager UI.

   

You may directly access the Resource Manager UI with https://<HDInsightClusterName>.azurehdinsight.net/yarnui/hn/cluster.

You see a list of jobs, along with their current state. In the screenshot, there's one job currently running:

To manually kill that running application, execute the following command from the SSH shell:

yarn application -kill <application_id>

For example:

yarn application -kill "application_1499348398273_0003"

Getting stuck in safe mode

When you scale down a cluster, HDInsight uses Apache Ambari management interfaces to first decommission the extra worker nodes. The nodes replicate their HDFS blocks to other online worker nodes. After that, HDInsight safely scales the cluster down. HDFS goes into safe mode during the scaling operation. HDFS is supposed to come out once the scaling is finished. In some cases, however, HDFS gets stuck in safe mode during a scaling operation because of file block under-replication.

By default, HDFS is configured with a dfs.replication setting of 1, which controls how many copies of each file block are available. Each copy of a file block is stored on a different node of the cluster.

When the expected number of block copies aren't available, HDFS enters safe mode and Ambari generates alerts. HDFS may enter safe mode for a scaling operation. The cluster may get stuck in safe mode if the required number of nodes aren't detected for replication.

Example errors when safe mode is turned on

org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/hive/819c215c-6d87-4311-97c8-4f0b9d2adcf0. Name node is in safe mode.

org.apache.http.conn.HttpHostConnectException: Connect to active-headnode-name.servername.internal.cloudapp.net:10001 [active-headnode-name.servername. internal.cloudapp.net/1.1.1.1] failed: Connection refused

You can review the name node logs from the /var/log/hadoop/hdfs/ folder, near the time when the cluster was scaled, to see when it entered safe mode. The log files are named Hadoop-hdfs-namenode-<active-headnode-name>.*.

The root cause was that Hive depends on temporary files in HDFS while running queries. When HDFS enters safe mode, Hive can't run queries because it can't write to HDFS. Temp files in HDFS are located in the local drive mounted to the individual worker node VMs. The files are replicated among other worker nodes at three replicas, minimum.

How to prevent HDInsight from getting stuck in safe mode

There are several ways to prevent HDInsight from being left in safe mode:

• Stop all Hive jobs before scaling down HDInsight. Alternately, schedule the scale down process to avoid conflicting with running Hive jobs.
• Manually clean up Hive's scratch tmp directory files in HDFS before scaling down.
• Only scale down HDInsight to three worker nodes, minimum. Avoid going as low as one worker node.
• Run the command to leave safe mode, if needed.

The following sections describe these options.

Stop all Hive jobs

Stop all Hive jobs before scaling down to one worker node. If your workload is scheduled, then execute your scale-down after Hive work is done.

Stop the Hive jobs before scaling, helps minimize the number of scratch files in the tmp folder (if any).

Manually clean up Hive's scratch files

If Hive has left behind temporary files, then you can manually clean up those files before scaling down to avoid safe mode.

1. Check which location is being used for Hive temporary files by looking at the hive.exec.scratchdir configuration property. This parameter is set within /etc/hive/conf/hive-site.xml:

   <property>
       <name>hive.exec.scratchdir</name>
       <value>hdfs://mycluster/tmp/hive</value>
   </property>
   

2. Stop Hive services and be sure all queries and jobs are completed.

3. List the contents of the scratch directory found above, hdfs://mycluster/tmp/hive/ to see if it contains any files:

   hadoop fs -ls -R hdfs://mycluster/tmp/hive/hive
   

   Here is a sample output when files exist:

   sshuser@scalin:~$ hadoop fs -ls -R hdfs://mycluster/tmp/hive/hive
   drwx------   - hive hdfs          0 2017-07-06 13:40 hdfs://mycluster/tmp/hive/hive/4f3f4253-e6d0-42ac-88bc-90f0ea03602c
   drwx------   - hive hdfs          0 2017-07-06 13:40 hdfs://mycluster/tmp/hive/hive/4f3f4253-e6d0-42ac-88bc-90f0ea03602c/_tmp_space.db
   -rw-r--r--   3 hive hdfs         27 2017-07-06 13:40 hdfs://mycluster/tmp/hive/hive/4f3f4253-e6d0-42ac-88bc-90f0ea03602c/inuse.info
   -rw-r--r--   3 hive hdfs          0 2017-07-06 13:40 hdfs://mycluster/tmp/hive/hive/4f3f4253-e6d0-42ac-88bc-90f0ea03602c/inuse.lck
   drwx------   - hive hdfs          0 2017-07-06 20:30 hdfs://mycluster/tmp/hive/hive/c108f1c2-453e-400f-ac3e-e3a9b0d22699
   -rw-r--r--   3 hive hdfs         26 2017-07-06 20:30 hdfs://mycluster/tmp/hive/hive/c108f1c2-453e-400f-ac3e-e3a9b0d22699/inuse.info
   

4. If you know Hive is done with these files, you can remove them. Be sure that Hive doesn't have any queries running by looking in the Yarn Resource Manager UI page.

   Example command line to remove files from HDFS:

   hadoop fs -rm -r -skipTrash hdfs://mycluster/tmp/hive/
   

Scale HDInsight to three or more worker nodes

If your clusters get stuck in safe mode frequently when scaling down to fewer than three worker nodes, then keep at least three worker nodes.

Having three worker nodes is more costly than scaling down to only one worker node. However, this action prevents your cluster from getting stuck in safe mode.

Scale HDInsight down to one worker node

Even when the cluster is scaled down to one node, worker node 0 still survive. Worker node 0 can never be decommissioned.

Run the command to leave safe mode

The final option is to execute the leave safe mode command. If HDFS entered safe mode because of Hive file under-replication, execute the following command to leave safe mode:

hdfs dfsadmin -D 'fs.default.name=hdfs://mycluster/' -safemode leave

Scale down an Apache HBase cluster

Region servers are automatically balanced within a few minutes after completing a scaling operation. To manually balance region servers, complete the following steps:

1. Connect to the HDInsight cluster using SSH. For more information, see Use SSH with HDInsight.

2. Start the HBase shell:

   hbase shell
   

3. Use the following command to manually balance the region servers:

   balancer
   

Next steps

• Automatically scale Azure HDInsight clusters

For specific information on scaling your HDInsight cluster, see:

• Manage Apache Hadoop clusters in HDInsight by using the Azure portal
• Manage Apache Hadoop clusters in HDInsight by using Azure CLI