Kafka Datasource

This section will demonstrate how to connect to a Kafka cluster, using the Kafka Datasource object.

Creating a Kafka Datasource

  1. To create a new Kafka Datasource object, in the I/O Model panel right-click on the Connector object to which you want to add the Kafka Datasource and select Admin  New  External Interfaces  Datasource from the context menu.

  2. In the Create Object Wizard, give a name to the object.

    Creating a new Kafka Datasource object - Common Section
    Figure 1. Creating a new Kafka Datasource object - Common Section
  3. Click Next to move on.

  4. In the Server Type section of the wizard, select 'Kafka' from the drop-down selection.

  5. Enter a Subscriber Name.

    Creating a new Kafka Datasource object - Server Type
    Figure 2. Creating a new Kafka Datasource object - Server Type Section
  6. If this Datasource is meant to receive data from the Kafka broker(s), it needs to be configured as a Consumer. To do so …​

    1. Go to the Consumer subsection.

    2. For the Bootstrap Servers property enter one or more <HOST>[:<PORT>][,] entries. Multiple entries can be provided, separated by commas. The Kafka default port is 9092.

    3. For Group ID or Static Group Instance ID enter a string which identifies the group to which this consumer belongs. The group is formed by consumers sharing the same Group ID.

      Creating a new Kafka Datasource object - Consumer Configuration
      Figure 3. Creating a new Kafka Datasource object - Consumer Configuration
  7. If this Datasource is meant to send data to the Kafka broker(s), it needs to be configured as a Producer. To do so …​

    1. Go to the Producer subsection.

    2. For the Bootstrap Servers property enter one or more <HOST>[:<PORT>][,] entries. Multiple entries can be provided, separated by commas. The Kafka default port is 9092.

      Creating a new Kafka Datasource object - Producer Configuration
      Figure 4. Creating a new Kafka Datasource object - Producer Configuration
  8. Click Create to close the wizard and create the Datasource object.

Subscribing to a Kafka Topic

To subscribe to a Kafka topic, IO-Item objects are used.

  1. To create a new IO-Item, right-click on the Kafka Datasource object in the I/O Model and select Admin  New  External Interfaces  IO-Item.

  2. In the Common section of the Create Object Wizard, enter an Object Name for the new IO-Item.

  3. For the Item ID property, provide the name of the Kafka to which this IO-Item should subscribe. If the topic name begins with '^'' then it becomes a pattern which allows a single IO-Item to subscribe to multiple topics. Received messages are forwarded to all IO-Items whose topics match the topic of the message.

    The Consumer IO-Item - Common Section
    Figure 5. The Consumer IO-Item - Common Section
  4. Next, go to the Item Type section and select 'Kafka' from the drop-down list.

  5. For Client Type select 'Consumer'.

    The Consumer IO-Item - Client Type Section
    Figure 6. The Consumer IO-Item - Client Type Section
  6. Click Create to close the wizard and create the IO-Item object.

Sending Messages to a Kafka Topic

To send messages to a Kafka topic, IO-Item objects are used.

  1. To create a new IO-Item, right-click on the Kafka Datasource object in the I/O Model and select Admin  New  External Interfaces  IO-Item.

  2. In the Common section of the Create Object Wizard, enter an Object Name for the new IO-Item.

  3. For the Item ID property provide the name of the Kafka topic.

    The Producer IO-Item - Common Section
    Figure 7. The Producer IO-Item - Common Section
  4. Next, go to the Item Type section, select 'Kafka' from the drop-down list.

  5. Enter the Kafka Partition if this IO-Item has to send messages to a specific partition only.

  6. If a Kafka Key is to be included in messages sent via this IO_item, provide it in the Key property.

  7. For Client Type select 'Producer'.

    The Producer IO-Item - Client Type Section
    Figure 8. The Producer IO-Item - Client Type Section
  8. Click Create to close the wizard and create the IO-Item object.