Overview of enterprise security in Azure HDInsight

Azure HDInsight offers a number of methods to address your enterprise security needs. Most of these solutions aren't activated by default. This flexibility allows you to choose the security features that are most important to you and helps you to avoid paying for features that you don't want. This flexibility also means it's your responsibility to make sure correct solutions are enabled for your setup and environment.

This article looks at security solutions by dividing security solutions into four traditional security pillars: perimeter security, authentication, authorization, and encryption.

This article also introduces the Azure HDInsight Enterprise Security Package (ESP), which provides Active Directory-based authentication, multi-user support, and role-based access control for HDInsight clusters.

Enterprise security pillars

One way of looking at enterprise security divides security solutions into four main groups based on the type of control. These groups are also called security pillars and are the following types: perimeter security, authentication, authorization, and encryption.

Perimeter security

Perimeter security in HDInsight is achieved through virtual networks. An enterprise admin can create a cluster inside a virtual network (VNET) and use network security groups (NSG) to restrict access to the virtual network. Only the allowed IP addresses in the inbound NSG rules can communicate with the HDInsight cluster. This configuration provides perimeter security.

All clusters deployed in a VNET will also have a private endpoint. The endpoint resolves to a private IP inside the VNET for private HTTP access to the cluster gateways.

Authentication

Enterprise Security Package from HDInsight provides Active Directory-based authentication, multi-user support, and role-based access control. The Active Directory integration is achieved through the use of Microsoft Entra Domain Services. With these capabilities, you can create an HDInsight cluster joined to an Active Directory domain. Then configure a list of employees from the enterprise who can authenticate to the cluster.

With this setup, enterprise employees can sign in to the cluster nodes by using their domain credentials. They can also use their domain credentials to authenticate with other approved endpoints. Like Apache Ambari Views, ODBC, JDBC, PowerShell, and REST APIs to interact with the cluster.

Authorization

A best practice most enterprises follow is making sure that not every employee has full access to all enterprise resources. Likewise, the admin can define role-based access control policies for the cluster resources. This action is only available in the ESP clusters.

The Hadoop admin can configure role-based access control (RBAC). The configurations secure Apache Hive, HBase, and Kafka with Apache Ranger plugins. Configuring RBAC policies allows you to associate permissions with a role in the organization. This layer of abstraction makes it easier to ensure people have only the permissions needed to do their work responsibilities. Ranger also allows you to audit the data access of employees and any changes done to access control policies.

For example, the admin can configure Apache Ranger to set access control policies for Hive. This functionality ensures row-level and column-level filtering (data masking). And filters the sensitive data from unauthorized users.

Auditing

Auditing cluster resource access is necessary to track unauthorized or unintentional access of the resources. It's as important as protecting the cluster resources from unauthorized access.

The admin can view and report all access to the HDInsight cluster resources and data. The admin can view and report changes to the access control policies.

To access Apache Ranger and Ambari audit logs, and ssh access logs, enable Azure Monitor and view the tables that provide auditing records.

Encryption

Protecting data is important for meeting organizational security and compliance requirements. Along with restricting access to data from unauthorized employees, you should encrypt it.

HDInsight supports data encryption at rest with both platform managed and customer managed keys. Encryption of data in transit is handled with both TLS and IPSec. See Encryption in transit for Azure HDInsight for more information.

Compliance

Azure compliance offerings are based on various types of assurances, including formal certifications. Also, attestations, validations, and authorizations. Assessments produced by independent third-party auditing firms. Contractual amendments, self-assessments, and customer guidance documents produced by Microsoft. For HDInsight compliance information, see the Microsoft Trust Center.

Shared responsibility model

The following image summarizes the major system security areas and the security solutions that are available to you in each. It also highlights which security areas are your responsibility as a customer. And which areas are the responsibility of HDInsight as the service provider.

The following table provides links to resources for each type of security solution.

Next steps

• Plan for HDInsight clusters with ESP
• Configure HDInsight clusters with ESP
• Manage HDInsight clusters with ESP