Consuming Azure Event Hubs Events With Kafka Connect

So last week I was in a rush to find a fast and easy way to consume events from Azure Event Hubs and send them to a Kafka topic.

After googling a bit I found this project: Kafka Connect Azure IoT Hub. Yes the name of the project can be misleading, but since IoT Hub is a service which relies on Event Hubs and also taking a close look to the code showed that it uses the Event Hubs client for java, I decided to give it a try.

I will asume not only that you have working knowledge with Event Hubs, but also that you have an instance deployed, plus a working Kafka and Kafka Connect (Distributed mode) setup.

Let’s kick it:

1. Download the Kafka Connect Azure IoT Hub

Download the Kafka Connect Azure IoT Hub 0.6 jar and copy the file in the Kafka installation libs folder (usually under KAFKA_HOME/libs).

Be sure to start Zookeper, Kafka and Kafka connect.

2. Create a connect-eventhub-source.json file

Update the following json and save it as connect-eventhub-source.json.

{
    "name": "eventhub-source",
    "config": {
        "connector.class": "com.microsoft.azure.iot.kafka.connect.IotHubSourceConnector",
        "tasks.max": "[Number of task == Number Event Hub Partitions]",
        "Kafka.Topic": "[Target Kafka Topic]",
        "IotHub.EventHubCompatibleName": "[Event Hubs Name]",
        "IotHub.EventHubCompatibleEndpoint": "sb://[Event Hubs Namespace].servicebus.windows.net/",
        "IotHub.AccessKeyName": "[Access key name for the Event Hub]",
        "IotHub.AccessKeyValue": "[Access key value for the Event Hub]",
        "IotHub.ConsumerGroup": "[Consumer group (Can use $Default)]",
        "IotHub.Partitions": "[Number of Event Hub Partitions]",
        "IotHub.StartTime": "",
        "IotHub.Offsets": "",
        "BatchSize": "100"
    }
}

I’ve submited a pull request to fix some of the descriptions you’ll find for the fields here

3. Post the configuration to the Kafka Connect endpoint

Assuming your Kafka Connect is running on localhost and listening to the default port 8083 execute the followinmg command:

curl -H "Content-Type: application/json" -d @connect-eventhub-source.json -X POST http://127.0.0.1:8083/connectors

After a while you should start receiving the events in the Kafka Topic you configured.

Hope it helps!

• ←Previous
• Next→