Filling History Gaps for History Transfer Controllers

History transfer processes which employ History Transfer Controllers consist of two steps: one where the historical data is brought from the Source Historian system into the inmation MongoDB, and the second where this data is deloyed to the Target Historian system. On both steps history gaps can occur.

Depending on where the history gap occured, either only the transfer to the Target Historian system needs to be repeated or the whole transfer process.

Checking Data Availability

In the I/O Model, select the History Transfer Controller object.

1 In the Property Panel, expand the Transporter Configuration section and note the settings for the Datasource and the Object Name property.

2 In the I/O Model, navigate to the object specified in the Datasource property and find the History Transporter object with the name as specified in the Object Name property.

Except for the _Statistics Variable Group, the tree structure underneath the History Transporter object corresponds to the tags specified in the Tag Configuration table property of the History Transfer Controller’s Transporter Configuration section where we just identified the History Transporter object.

3 Right-click on the first tag object and select Add item(s) to  History Grid  Time period …​

4 In the Time Period Setting, specify the relevant time period and click Apply.

5 For each other tag objects, select it in the I/O Model and drag it onto the History Grid Display to add the object to the grid.

6 In the History Grid Display, check if the expected data is available.

If all data is shown as expected, you can skip the next step and continue with the Refetching step. If <No Value> is shown in the Value …​ columns, the data is not available in the system’s MongoDB and needs to be backfilled first.

Backfilling

Backfilling re-imports historical data from the Source Historian into the system’s MongoDB.

This task is best executed by a separate history transfer process, for which an extra History Transporter object is required. While in general no History Transporter objects should be created directly (but only through History Transfer Controller or History Transport Manager objects), backfilling is an exception to this rule.

Creating a History Transporter Object

To create a new History Transporter Object …​

1 Connect DataStudio to the Core on the "Transferring" machine.

2 In the I/O Model, right-click the Connector object for the "Source Historian" and select Admin  New  History  History Transporter.

3 In the Create Object wizard, enter a Name.
In this example the object is called Backfill History Transporter.

4 Click Create to close the Create Object wizard and create the new History Transporter object in the I/O Model. It should be a sibling object of the History Transfer Controller or History Transport Manager for which it should backfill the history gap.

5 For the next steps, select the newly created History Transporter object in the I/O Model to display its properties in the Properties Panel.

Matching the Alternative Interface Configuration

If the original History Transport Mangager object has any properties of the Alternative Interface property compound configured, set these properties on the "Backfill" History Transporter object to the same settings.

Tag Specification

The tags for which history gaps should be filled, need to be specified in the Tag Configuration table property in the Common property compound of the "Backfill" History Transporter object. The easiest way to do this is to copy this information from the same table of the History Transfer Controller object and make some adjustments. Before these two steps are addressed in detail in later sections, the next section provides the essential background information.

Background Information

In the following table provides a brief overview over the roles of the columns in the Tag Configuration table:

Table 1. The Essential Columns of the Tag Configuration Table
Column Function

ExternalID

holds the Full Path to the tag underneath the DataSource object representing the Source Historian system

NodeName

specifies the path to the tag object in the transferring inmation system (without the object name itself)

ObjectName

specifies the name of the tag object in the transferring system

The entries for the NodeName and ObjectName columns together identify the tag object in the transferring system.

On the "Backfill" History Transporter object, the entries in the NodeName column need to point at the node/folder where the tag objects for the original history transfer process have been created.

Copying the Specification of the Relevant Tags

As all the tags of the original history transfer process are already specified in the responsible Tag Configuration table of the History Transer Controller object, copying the specification from there to the Tag Configuration table of the "Backfill" History Transporter provides a good starting point for the specification of the relevant tags for the backfill history transfer process.

To do this …​

1 Select this History Transfer Controller object in the I/O Model.

2 In the Property Panel, open the Transporter Configuration > Tag configuration table property by clicking on table icon, and move it to a place on the screen where it neither covers the I/O Model nor the Properties Panel.

3 Now select the "Backfill" History Transporter object in the I/O model and open the Tag configuration table property for this object, too.

4 For each relevant tag …​

a Find the tag in the table for the "Original" History Transfer Controller object.

b Right click on the far left-hand side of the row for the tag and select Copy from the context menu.

c Go to the table for the "Backfill" History Transporter object.

d Right click on the far left-hand side of the next empty row and select Paste from the context menu.

5 When all relevant tags have been added to the Tag Configuration table of the "Backfilling" History Transporter object, the table for the "Original" History Transfer Controller object can be closed by clicking OK.

Adjusting the NodeName Column on the "Backfill" History Transporter

The copied values for the NodeName column specify a relative path underneath the object to which the respective Tag Configuration table property belongs. If the backfill history transfer process would be started in the current state of the configuration, new object tags would be created underneath the the "Backfill" History Transporter object instead of the original tag objects being backfilled. The following steps show how the values in the NodeName column need to be adjusted so that they point at the tag objects used by the original history transfer process.

1 To identify the path to tag objects of the original history transfer objects …​

  1. In the I/O Model, select the History Transfer Controller Object for the history transfer process which needs backfilling and expand the Transporter Configuration compound in the Properties Panel.

  2. Take note of the Datasource [A], Folder Hierarchy [B] and Object Name [C] properties.

  3. In the I/O Model, locate the specified Datasource object [A] with the branch specified in the Folder Hierarchy property [B] underneath it. In this example the Folder Hirarchy property is empty, therefore the screenshot shows no corresponding branch in the I/O Model.

  4. At the bottom of this branch, you should find the History Transporter object [C] with the name specified in the Object Name property.

  5. Right-click on this History Transporter object and from the context menu select Copy full path.

2 In the Tag Configuration table for the "Backfilling" History Transporter object (which is still open), for each row, double-click on the NodeName column, place the cursor on the far left, before the existing entry and press CTRL-V to paste the copied path in front of this entry. Also add a slash ('/') between the copied path and the original entry.

3 When the entries in the NodeName column have been adjusted for all rows, click OK to close the Tag Configuration table.

Timespan Specification

The timespan for which datapoints are to be transferred for the specified tags is defined in the Depth property in the Common section of the "backfill" history transporter object.

Starttime and endtime are separated by a colon and both points in time are specified in relation to <now> which is symbolized by '*'. For example, a Depth of '*-1d2h :*-23h' would mean the three hours from 26 hours ago to 23 hours ago.

Disabling the Transfer of Up-To-Date Data

Since up-to-date data is not relevant for filling gaps in the historical data, unselect the checkbox for the Transfer UTD property.

Starting the Backfill Process

To start the backfill history transfer, set the Processing Mode property of the the History Transporter object to Continuous and click Apply for the Property Panel.

Expand the Diagnostics property compount of the Backfill History Transporter object to see the backfill at work. As soon as the History Progress bar shows 100% all data points for the specified tags and timespan have been sent to the target historian.

The backfill history transfer will transfer all datapoints which exist on the Source Historian system for the specified tags and in the specified timespan. This may result in duplicate data on the Target Historian system unless it provides some functionality to prevent this.

Finishing the Backfill Process

When the backfill history transfer process has finished, set the Processing Mode property of the the History Transporter object to Configuration and click Apply for the Property Panel.

Optimization Options

The History Transporter object offers a lot of options which can be used to optimize the efficiency of the backfill history transfer process as well as to control the burden this process creates for the system. These options are addressed in detail on the page regarding the Advanced History Transporter Options.

Refetching

Refetching transfers the historical data from the inmation MongoDB to the Target Historian system. The refetching process is executed by the History Transfer Controller object.

Tag Specification

To specify the tags which are to be refetched from the inmation MongDB to the Target Historian …​

1 In the I/O Model, select the History Transfer Controller object.

2 In the Property Panel, expand the Transporter Configuration section in question.

3 Find the Tag Configuration table property and click on the corresponding table icon to open the Table Property Grid.

4 In the row for each relevant tag, select the checkbox in the Refetch column.

5 When finished, click OK to close the Table Property Grid.

Alternatively, to select all tags in the Tag Configuration table properties of all Object Configuration and Transporter Configuration sections, the Refetch All checkbox in the Common section of the History Transfer Controller object can be selected. Selecting the Refetch Disable All checkbox unselects all tags in all …​ Configuration sections.

Timespan Specification

The timespan for which the datapoints of the specified tags shall be refetched, is specified in the Refetch Period of the History Transfer Controller object. For example, set this to *-2m:*-1m to specify the previous month.

Running the Refetch Process

If the Refetch Period is set, the execution of the refetch process starts as soon as the Apply is clicked for the Property Panel.