esi-mongo

esi-mongo provides means to retrieve inmation data directly from MongoDB

Dependencies

library version inmation core library

mongo

1.2.3

yes

io

5.3.0

yes

esi-string

1.72.0

yes

esi-tool

1.0.5

yes

Documentation

HIST_REMOVETIMESPANALL(self,connstr,dbname,collname,tstart,tend,batch_size,bulk_size)

Removing MongoDB data for selective time periods, for all objects

Parameters

connstr

MongoDB instance connection string

dbname

database name

collname

collection name

tstart

starttime for deletion (PosixMS)

tend

endtime for deletion (PosixMS)

batch_size

size of a single remove bulk, default 50.

bulk_size

size of a single bulk operation, default 10000.

HIST_REMOVETIMESPAN(self,connstr,dbname,collname,tstart,tend,prop_table,batch_size,bulk_size)

Removing MongoDB data for selective time periods and specific objects

Parameters

connstr

MongoDB instance connection string

dbname

database name

collname

collection name

tstart

starttime for deletion (PosixMS)

tend

endtime for deletion (PosixMS)

prop_table

table of property IDs (ItemValue)

batch_size

size of a single remove bulk, default 50.

bulk_size

size of a single bulk operation, default 10000.

HIST_DUMPTIMESPANALL(self,connstr,dbname,collname,tstart,tend,dump_path,batch_size)

Dump / Export MongoDB data for selective time periods, for all objects

Parameters

connstr

MongoDB instance connection string

dbname

database name

collname

collection name

tstart

starttime for deletion (PosixMS)

tend

endtime for deletion (PosixMS)

dump_path

export folder for data

batch_size

size of a single remove bulk, default 50.

HIST_DUMPTIMESPAN(self,connstr,dbname,collname,tstart,tend,prop_table,dump_path,batch_size)

Dump / Export MongoDB data for selective time periods and specific objects

Parameters

connstr

MongoDB instance connection string

dbname

database name

collname

collection name

tstart

starttime for deletion (PosixMS)

tend

endtime for deletion (PosixMS)

prop_table

table of property IDs (ItemValue)

dump_path

export folder for data

batch_size

size of a single remove bulk, default 50.

HIST_COUNT_DOCS(self,connstr,dbname,collname,starttime,endtime,chunkSize,_cb_state,maxChunks)

Create diagnostics output to show content of timeseries collection (documents per month)

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

starttime

starttime for deletion (PosixMS)

endtime

endtime for deletion (PosixMS)

chunkSize

Maximum amount of documents that is returned with each query.

_cb_state
maxChunks

Can be used for testing, to limit the number of queries by a hard limit

Usage

local MON = require 'esi-mongo'
local STR = require 'esi-string'
local VAR = require 'esi-variables'
local gt = syslib.gettime
local repo = syslib.model.codes.RepoStoreName.STORE_RAW_HISTORY
local res = {}
res.startt = syslib.currenttime()
res.connstr = syslib.getmongoconnectionstring(repo)
local client = MON:_ensureClient(res.connstr)
local function reportState(state)
local percDone = 0
if type(state.queryIteration)=="number" and type(state.maxQueries)=="number" then
percDone = 100 * state.queryIteration / state.maxQueries
end
VAR:SET("state",state)
VAR:SET("completed%",percDone)
end
local chunkSize = 1000
res.result = MON:HIST_COUNT_DOCS(client, "History", "rawdata", gt("2000-01-01T00:00:00.000Z"), gt("2030-01-01T00:00:00.000Z"), chunkSize, reportState)
res.durationMS = syslib.currenttime() - res.startt
return STR:STRING(res)

HIST_PURGE_DOCS(self,connstr,dbname,collname,starttime,endtime,chunkSize,_cb_state,maxChunks,deleteTresholdMs)

Remove hour documents from history collection, that are older (smaller) then threshold.

The method works by iterating the timeseries collection, rather than quering content for all tags. On very large collections (> 1 TB), this can

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

starttime

starttime for deletion (PosixMS)

endtime

endtime for deletion (PosixMS)

chunkSize

Maximum amount of documents that is returned with each query.

_cb_state

Callback method, than can be used to report progress on the delete operation.

maxChunks
deleteTresholdMs

Can be used for testing, to limit the number of queries by a hard limi

Usage

see above

BULK(self,connstr,dbname,collname,queryType,queryStrList,documentStrList,options,prefs,callback)

Execute a bulk operation with multiple queries of one type

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryType

specify type of bulk query (LIB.QUERYTYPE.FIND, .COUNT, .AGGREGATE, .COMMAND, .UPDATE, .UPDATE, .UPDATE, .REMOVE, .INSERT)

queryStrList

table with strings, each string represents a mongodb query that is executed in the bulk operation

documentStrList

used for update queries: table with strings, each string represents a mongodb document that is for updating

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

FIND(self,connstr,dbname,collname,queryStr,options,prefs,cb)

Query MongoDB with FIND

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

UPDATE(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)

Query MongoDB with UPDATE

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

documentStr

valid JSON document, used to update the query result returned by queryStr

UPDATEONE(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)

Query MongoDB with UPDATEONE

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

documentStr

valid JSON document, used to update the query result returned by queryStr

UPDATEMANY(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)

Query MongoDB with UPDATEMANY

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

documentStr

valid JSON document, used to update the query result returned by queryStr

REMOVE(self,connstr,dbname,collname,queryStr,options,prefs,cb)

Query MongoDB with REMOVE

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

REMOVEONE(self,connstr,dbname,collname,queryStr,options,prefs,cb)

Query MongoDB with REMOVEONE

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

REMOVEMANY(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)

Query MongoDB with REMOVEMANY

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

INSERT(self,connstr,dbname,collname,queryStr,options,prefs,cb)

Query MongoDB with INSERT

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON query string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

COMMAND_FIND(self,connstr,dbname,collname,filter,projection,sort,limit,options,prefs,cb)

Query MongoDB with Command (FIND)

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

filter

valid JSON filter string

projection

valid JSON projection string

sort

specify list of fields with sort order

limit

specify maximum number of returned documents

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

COMMAND(self,connstr,dbname,commandStr,options,prefs,cb)

Query MongoDB with Command

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

commandStr

valid JSON command string

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

AGGREGATE(self,connstr,dbname,collname,pipeline,options,prefs,cb)

Query MongoDB aggregation pipeline

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

pipeline

valid JSON string that described MongoDB aggregation pipeline

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

cb

per-document callback for result processing

COUNT(self,connstr,dbname,collname,queryStr,options,prefs)

Query MongoDB with count

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

queryStr

valid JSON string that described MongoDB aggregation pipeline

options

table used for setting the options in a query.

prefs

table used for setting the prefs in a query.

CURRENTOP(self,connstr,all)

Wrapper around db.currentOp() command

requires permissions for admin DB, see MongoDB documentation

Parameters

connstr

MongoDB instance connection string

all

Return all operations, or just a top x

SERVERSTATUS(self,connstr)

Wrapper around db.serverStatus() command

Parameters

connstr

MongoDB instance connection string

DBSTATS(self,connstr,dbname)

Wrapper around db.stats() command

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

COLLECTIONSTATS(self,connstr,dbname,collname)

Wrapper around db.collection.stats() command

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

collname

collection name, default rawdata

SHOWDBS(self,connstr)

Wrapper around the Lua Mongo getDatabaseNames

Parameters

connstr

MongoDB instance connection string

SHOWCOLLECTIONS(self,connstr,dbname)

Wrapper around the db.getCollectionNames() command

Parameters

connstr

MongoDB instance connection string

dbname

database name, default History

GETLOCALDBINFO(self,storeCode,includeDBs,excludeDBs,toVariable,scalingFactor)

Full statistics package in one method

Method retrieves ServerStatus, DBStats and CollectionStats Can include or exclude DBs to make execution faster

Parameters

storeCode

Specify syslib.model.codes.RepoStoreName to get MongoDB connection string

includeDBs

table with list of string, that are used to find all databases that will be included (rest will be excluded). Default: nil - all tables will be included.

excludeDBs

table with list of string, that are used to find all databases that will be excluded (rest will be included). Default: nil - no tables will be excluded.

toVariable

dump result to variable objects, default: nil - will not dump results

scalingFactor

Scaling factor for numeric values

PARSE_URL(connectionString)

PARSE_URL: Parse MongoDB connection in the Standard Connection String format

Returns Lua table with the properties username, password, hosts, dbname, parameters.

Note 1: Will throw error if parsing failed

Note 2: Does not support the DNS Seedlist Connection Format

Parameters

connstr

MongoDB instance connection string

Usage

-- Examples:
local test1 = "mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017,cluster0-shard-00-01-osrtv.mongodb.net:27017,cluster0-shard-00-02-osrtv.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true&w=majority"
local test2 = "mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017/test"
local test3 = "mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017"
local test4_fail1 = "mongodb://myuser@cluster0-shard-00-00-osrtv.mongodb.net:27017"
local test5_fail2 = "mongodb://cluster0-shard-00-00-osrtv.mongodb.net"

SET_STORES(self,connStr,authType,ignoreSystemLog,storeRootPath,noPasswordOverride)

Set the MongoDB connection properties for system object for custom datastores

Parameters

connectionString

Connection as connection string or or table with fields connection, username, password

authType

Specify authentication type as syslib.model.codes.MongoDBAuthentication (Default MONGAUTH_SCRAM_SHA_256).

ignoreSystemLog

If set to true the SystemLog system store will be ignored, and keep its old property values (Default nil).

storeRootPath

If set, the execution will try to set all children of a Data Store Group object, as specified by this path. If not set, the system object will be used.

noPasswordOverride

If set to true the method will check if password has been set before. If yes, an error will be created.

Usage

-- Example 1: Set all stores to localhost MongoDB, with no authentication
return (require 'esi-mongo'):SET_STORES({
hosts={"localhost:27017"},
username="",
password=""
}, syslib.model.codes.MongoDBAuthentication.MONGAUTH_NONE)

-- Example 2: Set all stores to PSA replicaset, with default authentication
return (require 'esi-mongo'):SET_STORES({
hosts={"mongo1:27017", "mongo2:27017", "mongo3:27019"},
username="so",
password="<PASSWORD>",
parameters = {replicaSet="rs0"}
})

-- Example 3: Set all stores to standalone MongoDB, with authentication
return (require 'esi-mongo'):SET_STORES("mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017")

CREATE_CUSTOM_STORES(self,corePath,namePrefix,ignoreSets,retentionTimeDays1,retentionTimeDays2,addInfStore2,localDS_hasLocal,localDS_hasMaster,localDS_hasMasterLocal)

Create custom datastores according to standard naming convention

Naming convention: {CorePath}/Data Stores/{namePrefix}

Custom datastores for all 3 datatypes will be created (Timeseries, Events, Production Tracking). Each type will have 2 stores (purged / non-purged).

Datastore sets will be created for timeseries stores.

When executed on Local-Core, Datastore sets will be created for 3 variants: Local / Master / storage on both Local and Master-Core.

Parameters

corePath

Specifis path to Master- or Local-Core

namePrefix

Specifies prefix for store name

ignoreSets

Set to false no Datastore sets will be created (default false)

retentionTimeDays1

Configure retention time for store 1 (non-Gxp). Default = 3650. If set to -1, purging is switched off.

retentionTimeDays2

Configure retention time for store 2 (GxP). Default = 3650. If set to -1, purging is switched off.

addInfStore2

Create a second infite store for store 2 (GxP), but only on Master-Core

localDS_hasLocal

Create Local-Core Data Store Set for storing data local

localDS_hasMaster

Create Local-Core Data Store Set for storing data in central

localDS_hasMasterLocal

Create Local-Core Data Store Set for storing data local and central

Usage

local MONGO = require "esi-mongo"
local corePath = syslib.getcorepath()
local prefix = "GLOBAL"
local ignoreSets = false
local retentionTimeDays1 = 1 * 365 -- 1 year
local retentionTimeDays2 = 1 * 365 -- 1 year
local addInfStore2 = true
--local localDS_hasLocal = true
--local localDS_hasMaster = true
local localDS_hasMasterLocal = true
MONGO:CREATE_CUSTOM_STORES(corePath, prefix, ignoreSets, retentionTimeDays1, retentionTimeDays2, addInfStore2, localDS_hasLocal, localDS_hasMaster, localDS_hasMasterLocal)