Quickstart: Stream data with Azure Event Hubs and Apache Kafka

1. Enable a system-assigned managed identity for the virtual machine. For more information about configuring managed identity on a VM, see Configure managed identities for Azure resources on a VM using the Azure portal. Managed identities for Azure resources provide Azure services with an automatically managed identity in Microsoft Entra ID. You can use this identity to authenticate to any service that supports Microsoft Entra authentication, without having credentials in your code.

   

2. Using the Access control page of the Event Hubs namespace you created, assign Azure Event Hubs Data Owner role to the VM's managed identity. Azure Event Hubs supports using Microsoft Entra ID to authorize requests to Event Hubs resources. With Microsoft Entra ID, you can use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, or an application service principal.

   1. In the Azure portal, navigate to your Event Hubs namespace. Go to "Access Control (IAM)" in the left navigation.

   2. Select + Add and select Add role assignment.

      

   3. In the Role tab, select Azure Event Hubs Data Owner, and select the Next button.

      

   4. In the Members tab, select the Managed Identity in the Assign access to section.

   5. Select the +Select members link.

   6. On the Select managed identities page, follow these steps:

      1. Select the Azure subscription that has the VM.

      2. For Managed identity, select Virtual machine

      3. Select your virtual machine's managed identity.

      4. Select Select at the bottom of the page.

         

   7. Select Review + Assign.

      

3. Restart the VM and sign in back to the VM for which you configured the managed identity.

4. Clone the Azure Event Hubs for Kafka repository.

5. Navigate to azure-event-hubs-for-kafka/tutorials/oauth/java/managedidentity/consumer.

6. Switch to the src/main/resources/ folder, and open consumer.config. Replace namespacename with the name of your Event Hubs namespace.

   bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
   security.protocol=SASL_SSL
   sasl.mechanism=OAUTHBEARER
   sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
   sasl.login.callback.handler.class=CustomAuthenticateCallbackHandler;
   

   Note

   You can find all the OAuth samples for Event Hubs for Kafka here.

7. Switch back to the Consumer folder where the pom.xml file is and, and run the consumer code and process events from event hub using your Kafka clients:

   mvn clean package
   mvn exec:java -Dexec.mainClass="TestConsumer"                                    
   

8. Launch another command prompt window, and navigate to azure-event-hubs-for-kafka/tutorials/oauth/java/managedidentity/producer.

9. Switch to the src/main/resources/ folder, and open producer.config. Replace mynamespace with the name of your Event Hubs namespace.

10. Switch back to the Producer folder where the pom.xml file is and, run the producer code and stream events into Event Hubs:

    mvn clean package
    mvn exec:java -Dexec.mainClass="TestProducer"                                    
    

    You should see messages about events sent in the producer window. Now, check the consumer app window to see the messages that it receives from the event hub.

    

1. Clone the Azure Event Hubs for Kafka repository.

2. Navigate to azure-event-hubs-for-kafka/quickstart/java/consumer.

3. Update the configuration details for the consumer in src/main/resources/consumer.config as follows:

   bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
   

   Important

   Replace {YOUR.EVENTHUBS.CONNECTION.STRING} with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see Get an Event Hubs connection string. Here's an example configuration: sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";

4. Run the consumer code and process events from event hub using your Kafka clients:

   mvn clean package
   mvn exec:java -Dexec.mainClass="TestConsumer"
   

5. Navigate to azure-event-hubs-for-kafka/quickstart/java/producer.

6. Update the configuration details for the producer in src/main/resources/producer.config as follows:

   bootstrap.servers=NAMESPACENAME.servicebus.windows.net:9093
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
   

   Important

   Replace {YOUR.EVENTHUBS.CONNECTION.STRING} with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see Get an Event Hubs connection string. Here's an example configuration: sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";

7. Run the producer code and stream events into Event Hubs:

   mvn clean package
   mvn exec:java -Dexec.mainClass="TestProducer"
   

If your Event Hubs Kafka cluster has events, you'll now start receiving them from the consumer.