History Transfer Controller with MQTT Publisher

The History Transfer Controller object can be configured to work with different kinds of sinks, including MQTT publisher.

For information regarding the general configuration, see the section on Using History Transfer Controllers in the History Transfer JumpStart.

This section will pick up where that example finished, with a configured History Transfer Controller object and linked History Sink object below a Connector.

We will also use the MQTT broker endpoint created in the previous Azure IoT Hub setup section.

Configuring History Transfer Controller to work with MQTT publisher

  1. Create a new History Transfer Controller below the Core or select the History Transfer Controller object created in the How to export history data using History Transfer Controller example. Go to the Sink Configuration property compound in the Object Properties panel. If you are cretaing a new History Transfer Controller object, enter a name for the HIstory SInk and the path to a parent Connector. Under Configuration, change the History Sink Interface to "MQTT Publisher".

    Sink Configuration - Interface
    If you create a new HIstory Controller object, you will also need to add some tags with history data to the Object Configuration, as described in the How to export history data using History Transfer Controllersection of the documentation.
  2. Open the MQTT Publisher Parameters sub compound. The configuration entered here should match the IoT device parameters created in the Azure IoT Hub setup. Also an SAS token needs to be generated, which is used for authenticating our publisher. Enter the Topic, with a value of "devices/ + [ Device ID ] + /messages/events/" (device ID = “inmIoTDevice01”). For example:

    devices/inmIoTDevice01/messages/events/

    Enter the Quality Of Service as 1. This means the publisher will wait for confirmation from Broker that it received a message.

    Sink Configuration - MQTT Publisher Parameters
  3. Open the Connection Parameters sub-compound and enter the Client ID as the device ID (device ID = “inmIoTDevice01”). Enter the Azure Mqtt Broker Port: as 8883 and enter the Broker Address as [ Hub name ] + .azure-devices.net (hub name = “inmIoTHubFree”). For example:

    inmIoTHubFree.azure-devices.net
    Sink Configuration - Connection Parameters
  4. Open the Credentials sub-compound and specify the username in the format [ Hub name ] + .azure-devices.net/ + [ Device ID ] + /?api-version=2018-06-30. For example:

    inmIoTHubFree.azure-devices.net/inmIoTDevice01/?api-version=2018-06-30

    Specify the SAS token generated in the Azure IoT Hub setup earlier.

    Sink Configuration - Credentials
  5. Open the SSL/TLS Support sub-compound and enter the Certificate Authority File, as created in the Azure IoT Hub setup earlier.

    Sink Configuration - SSL/TLS Support
  6. To begin the history export immediately, go back to the Common property compound and change the Processing Mode from "Configuration" to "Operation" and click Apply to begin.

    History Transfer Controller - Processing Mode
  7. To check on the progress of the MQTT History Sink export, go to the Diagnostics property compound of the History Transfer Controller.

    • Process State will show you when the message be sent to History-Sink and file will be created

    • Recent Data Size will indicate how large the last file was

    • UTD-Values shows how many samples have been written

    The History Transfer Controller should also show a green light in the I/O Model.

    History Transfer Controller - Diagnostics
  8. Select the History Sink object in the I/O Model (this will be below the Connector object specified during the History Transfer Controller configuration). Open the Diagnostics property compound for the History Sink and check the status of the most recent export.

    History Sink Diagnostics
  9. The progress can also be followed by using Visual Studio Code and installing the IoT Azure Hub extension.

    MQTT message monitoring in Visual Studio Code

Smart Message Configuration

MQTT has a maximum message size of 256 KB. Depending on the amount of data you transfer, it can easily happen that the JSON messages get larger the 256 KB. It is possible to use the Smart Messages feature to split up the messages automatically. Using Smart Messages, you can also prepare different JSON formats.

  1. To do this, open the Sink Configuration property compound on the History Transfer Controller and go to Advanced Options  Transfer Processingand in the Smart Messages property enter "Flat JSON".

    Sink Configuration - Smart Messaging
  2. Click Apply and check Diagnostics to see when the first messages have been sent.

  3. If Using Visual Studio Code and the IoT Azure Hub extension,you can monitor the endpoint and confirm that messages are received.

    Visual Studio Code - Flat JSON Smart Messaging
The Flat JSON format is ideal for ingesting the data in Azure Time Series Insights.