Read Raw Historical Data
-
Can return Historical data points or a query strategy.
-
Returns by default a query strategy when the total number of data points exceeds three million.
-
Can be forced to return less data points by setting the 'query_count_limit'
-
Can be used to fetch historical data in reversed order by switching the start and end time.
-
Accepts a filter for server side filtering.
-
Accepts 'Match Duration' filters to perform server side duration calculations.
-
Required security permission - READ.
-
HTTP methods supported: GET, POST.
Description
This endpoint allows to read raw historical data for provided time period for one or more items.
Parameters
Name | Type | Optional | Description |
---|---|---|---|
start_time |
string |
no |
The start time (UTC) of the interval to retrieve data for. |
end_time |
string |
no |
The end time (UTC) of the interval to retrieve data for. |
identifier |
string |
no |
The path or ID of the object to be queried for historical information. |
filter |
string |
yes |
Filter query. Has to be a JSON (dictionary). In case of a HTTP GET the value of this URL query parameter must be Base64 encoded. |
fields |
string |
yes |
Comma separated string to customize which fields should be included in the response, by default 'i,p,v,q,t'. Provide 'ALL' to include all fields. |
bounds |
undefined |
yes |
A Boolean or number, which indicates bounding values should be returned, by default false. 0 / false = None, 1 / true = both start and end, 2 = only start, 3 = only end. |
query_count_limit |
integer |
yes |
The maximum number of data points per query. Optional, by default 3000000. |
query_index |
string |
yes |
The index of the query to retrieve historical data for. In case a query index is provided, the query will be executed directly, without determining a strategy first. |
insights |
boolean |
yes |
Indicates application insights should be included in the response, by default false. |
Filtering
The filter option can either be an Aggregation Pipeline with the supported stages '\(match' and '\)limit' or a match query. The syntax correspond respectively to the MongoDB’s Aggregation Pipeline or read operation query syntax.
Supported Pipeline Stages
Stage | Description |
---|---|
$match |
Filters the historical data points. |
$limit |
Limits the number of matching historical data points. The first n data points will be returned, where n is the specified limit. |
Supported query selectors are:
Comparison
Name | Description |
---|---|
$eq |
Matches values that are equal to a specified value. |
$gt |
Matches values that are greater than a specified value. |
$gte |
Matches values that are greater than or equal to a specified value. |
$in |
Matches any of the values specified in an array. |
$lt |
Matches values that are less than a specified value. |
$lte |
Matches values that are less than or equal to a specified value. |
$ne |
Matches all values that are not equal to a specified value. |
$nin |
Matches none of the values specified in an array. |
Query Strategy
By default read raw historical data returns a query strategy when the total number of data points exceeds three million. If "query_count_limit" is set to lower number of data points than requested data point number, response will also be a query strategy. To get the usual "v", "q", "t" response structure read raw historical data endpoint can be called providing the query index as a parameter for GET call or the query as JSON dictionary as a request body for POST call.
Duration calculations
With the field option 'd' it is possible to request the server to calculate the duration between each historical data point. The first duration is calculated from the interval start time, the last duration is calculated till the end time of the requested interval.
Interval Interval
Start End
+ +
40 | x |
30 | x |
20 | x |
10 | x |
+---------------------------------
<----> <----> <----> <----> <---->
d[1] d[2] d[3] d[4] d[5]
Match Duration
The 'Match Duration' feature gives the possibility to perform (server-side) duration calculations based on a match expression. To be able to calculate the duration for the first and last data point in the requested interval the first matching bounding values will automatically be fetched based on the provided Match duration options:
Name | Description |
---|---|
leading_bounding_time_span |
The maximum time span in milliseconds before the leading bounding value of the requested interval to search for a matching bounding value. |
leading_bounding_intervals_no |
The number of intervals to use to search for a matching leading bounding value. |
trailing_bounding_time_span |
The maximum time span in milliseconds after the trailing bounding value of the requested interval to search for a matching bounding value. |
trailing_bounding_intervals_no |
The number of intervals to use to search for a trailing leading bounding value. |
The following example shows how Match Durations can be defined in a 'ReadRawHistoricalDataQuery' / request body:
{
"md": [
{
"name": "MyDuration01",
"expression": {
"v": {
"$gt": 20
}
},
"leading_bounding_time_span": 15000,
"trailing_bounding_time_span": 15000,
"leading_bounding_intervals_no": 3,
"trailing_bounding_intervals_no": 3
}
]
}
For example, setting the 'leading_bounding_time_span' 15000 and the 'leading_bounding_intervals_no' to 3, means the logic searches for a matching leading bounding value in three steps starting from the timestamp of the leading bounding value of the requested interval.
matching
leading interval
bounding leading start
value bounding +
x value |
.<------>+
3000ms |
|
<------------------------------------->+---+
leading bounding time span
<----><----------><---------->
2000ms 5000ms 5000ms
(interval 3)(interval 2)(interval 1)
Data point 'x' is included in the response and used in the match duration calculation. Data point '.' is excluded from the response and match duration calculation.
Examples
The following example shows the output based on the Match Duration expression '{"v" : {"$gt": 20}}':
interval interval
start end
+ +
x 40 | x | x
30 | x |
* 20 | * | *
10 | * |
+--------------------------------+
<---------------------> <-----------> <------------>
d[1] d[2] d[3]
Data points indicated by ' * ' are included in the response but excluded from the match duration calculation.
Match Duration calculations can be used together with the filter option. The following example shows the output based on the Match Duration expression '{"v" : {"\(gt": 20}}' and filter '{"v" : {"\)gte": 20}}'
interval interval
start end
+ +
x 40 | x | x
30 | x |
* 20 | * | *
10 | . |
+--------------------------------+
<---------------------> <-----------> <------------>
d[1] d[2] d[3]
Post Execution
The Post execution feature makes it possible to execute any Lua logic server-side (in the Core) to process / transform the data returned by a 'ReadRawHistoricalData' query.
The following example shows how Post Execution functions can be defined in a
ReadRawHistoricalDataQuery
/ request body:
{
"post_execution": [
{
"lib": "CalcLib",
"func": "countOccurrences",
"farg": {}
}
]
}
The library (lib
), the function in the library (func
) and the argument passed to the function (farg
, see Function Argument below) are defined here.
The library defined in a Post Execution item must be available in the context of the Web API. An example library to count the number of historical data points per item can be found here
The implementation of the library
local lib = {}
function lib:countOccurrences(arg)
arg = arg or {}
local result = {}
local data = arg.data or {}
for _, item in ipairs(data.items) do
local count = 0
-- v can contain nil values, use t.
if type(item.t) == 'table' then
count = #item.t
end
table.insert(result, {
p = item.p,
v = count
})
end
return result
end
return lib
Function argument
The raw historical data query result gets passed to the function as part of the
arg
argument. The JSON representation is as follows:
{
"data": {
"query_index": "1",
"start_time": "2018-11-11T00:00:00.000Z",
"end_time": "2018-11-13T00:00:00.000Z",
"items": [
{
"p": "/System/Core/...",
"v": [],
"q": [],
"t": []
}
]
}
}
Response content type
Post Execution makes it also possible to return the data in a different format
then JSON. The example library
rrhd-post-exec-with-headers show
how to return a csv by defining the HTTP headers
'["Content-Type"] = "txt/csv"'
in the response.