Cloud Sink with Azure Event Hub
In this Jumpstart we will create a new Microsoft Azure Event Hub with Kafka support and configure it. We will then use the Cloud Sink object as a producer to send a message to Azure Event Hub using the Kafka protocol.
To complete the examples in this Jumpstart the following things are required:
|Azure account which is going to be used should be initially configured, meaning that there already should be a valid Azure subscription and all the necessary permissions to add and manage resources. There also should be at least one Resource Group. More information about Resource Groups is available on the official Microsoft Documentation webpage|
system:inmation installation (v1.54 and above) including DataStudio
Preparing Azure Infrastructure
Login to Azure Portal. Click on button
Find and add a new Event HubsFigure 1. Adding a new Event Hubs namespace
Make sure you have the following parameters set using the suggested values or use the below screenshot as a guide:
Name - This will be used during the configuration of the Cloud Sink object so make a note of it. In this example the following name was used: inmationKafkaTest
Pricing tier - Standard
Kafka protocol is not supported in Basic pricing tier. Please refer to the official Microsoft Documentation webpage for additional information
Figure 2. Event Hubs Namespace configuration
Enable Kafka - Checked
Click and wait until the resource is deployed
When Event Hub is up and running, select it under .
Create a new Event Hub ( ). Give the object a name and make note of it (this will also be used in the configuration of the Cloud Sink object and as a Kafka Topic). In this example Event Hub will be named inhub.Figure 3. Adding a new Event HubFigure 4. Configuring a new Event Hub
Select the Event Hub Namespace object you just created and find the connection string to configure under the RootManageSharedAccessKey Policy ( ). It should be similar to this:
Make a note of it for later when configuring the the Cloud Sink object.Figure 5. Obtaining Connection String
Download the cacert.pem certificate file available here
Simple copy-pasting may not work as some extra line-breaker characters might appear and corrupt the certificate. It is recommended to use browser Save as… command or use any download tool which can accept URL as a file source.
|In case of encountering any troubles, this official Microsoft Quickstart can be used as an alternative example of how to create a new Event Hub|
Cloud Sink configuration
In DataStudio create a new Cloud Sink object by selecting a Connector and right-clicking. Selectfrom the context menu. Give the object a name a click Create to create it in the I/O Model.
Select the Cloud Sink object and in the Object Properties panel, open the Configuration property section.
Under Kafka Producer Parameters enter for the Topic property, the Event Hub name from Preparing Azure Infrastructure: Step 6.
Table 1. Kafka Producer Parameters Parameter Value Reference Example
[ Event Hub name ]
inhubFigure 6. Cloud Sink Kafka configuration
Under Global Configuration Properties, enter the following values for the parameters:
Table 2. Global Configuration Properties Parameter Value Reference Example
[ Event Hubs Namespace name ] +
[ Connection string ]
Endpoint=sb://inmationKafkaTest.servicebus.windows.net/; SharedAccessKeyName=RootManageSharedAccessKey; SharedAccessKey=cHJpbWFyeSBvciBzZWNvbmRhcnkga2V5Ow==
Azure Kafka service is listening to port 9093Figure 7. Cloud Sink Global configuration
Import Certificate Authority file
Click Import File on property SSL Certificate Authority and select cacert.pem file downloaded in Preparing Azure Infrastructure: Step 8Figure 8. Certificate Authority propertyFigure 9. Selecting Certificate Authority file
If everything is done correctly, it should be possible to view certificate details by clicking ViewFigure 10. Certificate details
Click Apply in the Object Properties to save configuration.
Transferring data with Cloud Sink
To check that the interface is working, double-click on Cloud Sink object’s Faceplate and enter a string in the Write Value dialog.Figure 11. Writing a value to Cloud Sink Item Value
Now check the Azure Event Hub metric to see that the message has been captured.Figure 12. Azure Event Hub metric with the number of incoming messages
Alternatively you can install the Azure Event Hub Explorer extension for Visual Studio Code and check the result thereFigure 13. An incoming message reported by Azure Event Hub Explorer extension
After deleting a Cloud Sink object, some certificates might be left in the corresponding certificate stores. In order to remove old certificates run the following command in a console:
Executing this command will not affect any certificates, which are currently being used by other Cloud Sink objects