esi-mongo
esi-mongo provides means to retrieve inmation data directly from MongoDB
Dependencies
| library | version | inmation core library |
|---|---|---|
mongo |
1.2.3 |
yes |
io |
5.3.0 |
yes |
esi-string |
1.72.0 |
yes |
esi-tool |
1.0.5 |
yes |
Available functions
All functions have to be called according to the ESI standard, using colons, e.g. lib:FUNCTIONNAME(params)
Documentation
HIST_REMOVETIMESPANALL(self,connstr,dbname,collname,tstart,tend,batch_size,bulk_size)
Removing MongoDB data for selective time periods, for all objects
Usage see Backup and Restoration Actions
HIST_REMOVETIMESPAN(self,connstr,dbname,collname,tstart,tend,prop_table,batch_size,bulk_size)
Removing MongoDB data for selective time periods and specific objects
Usage see Backup and Restoration Actions
HIST_DUMPTIMESPANALL(self,connstr,dbname,collname,tstart,tend,dump_path,batch_size)
Dump / Export MongoDB data for selective time periods, for all objects
Usage see Backup and Restoration Actions
HIST_DUMPTIMESPAN(self,connstr,dbname,collname,tstart,tend,prop_table,dump_path,batch_size)
Dump / Export MongoDB data for selective time periods and specific objects
Usage see Backup and Restoration Actions
HIST_COUNT_DOCS(self,connstr,dbname,collname,starttime,endtime,chunkSize,_cb_state,maxChunks)
Create diagnostics output to show content of timeseries collection (documents per month)
Usage
local MON = require 'esi-mongo'
local STR = require 'esi-string'
local VAR = require 'esi-variables'
local gt = inmation.gettime
local repo = inmation.model.codes.RepoStoreName.STORE_RAW_HISTORY
local res = {}
res.startt = inmation.currenttime()
res.connstr = inmation.getmongoconnectionstring(repo)
local client = MON:_ensureClient(res.connstr)
local function reportState(state)
local percDone = 0
if type(state.queryIteration)=="number" and type(state.maxQueries)=="number" then
percDone = 100 * state.queryIteration / state.maxQueries
end
VAR:SET("state",state)
VAR:SET("completed%",percDone)
end
local chunkSize = 1000
res.result = MON:HIST_COUNT_DOCS(client, "History", "rawdata", gt("2000-01-01T00:00:00.000Z"), gt("2030-01-01T00:00:00.000Z"), chunkSize, reportState)
res.durationMS = inmation.currenttime() - res.startt
return STR:STRING(res)HIST_PURGE_DOCS(self,connstr,dbname,collname,starttime,endtime,chunkSize,_cb_state,maxChunks,deleteTresholdMs)
Remove hour documents from history collection, that are older (smaller) then threshold.
The method works by iterating the timeseries collection, rather than quering content for all tags. On very large collections (> 1 TB), this can
BULK(self,connstr,dbname,collname,queryType,queryStrList,documentStrList,options,prefs,callback)
Execute a bulk operation with multiple queries of one type
Parameters
queryType
specify type of bulk query (LIB.QUERYTYPE.FIND, .COUNT, .AGGREGATE, .COMMAND, .UPDATE, .UPDATE, .UPDATE, .REMOVE, .INSERT)
queryStrList
table with strings, each string represents a mongodb query that is executed in the bulk operation
UPDATE(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)
Query MongoDB with UPDATE
UPDATEONE(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)
Query MongoDB with UPDATEONE
UPDATEMANY(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)
Query MongoDB with UPDATEMANY
REMOVEONE(self,connstr,dbname,collname,queryStr,options,prefs,cb)
Query MongoDB with REMOVEONE
REMOVEMANY(self,connstr,dbname,collname,queryStr,options,prefs,cb,documentStr)
Query MongoDB with REMOVEMANY
COMMAND_FIND(self,connstr,dbname,collname,filter,projection,sort,limit,options,prefs,cb)
Query MongoDB with Command (FIND)
AGGREGATE(self,connstr,dbname,collname,pipeline,options,prefs,cb)
Query MongoDB aggregation pipeline
CURRENTOP(self,connstr,all)
Wrapper around db.currentOp() command
requires permissions for admin DB, see MongoDB documentation
GETLOCALDBINFO(self,storeCode,includeDBs,excludeDBs,toVariable,scalingFactor)
Full statistics package in one method
Method retrieves ServerStatus, DBStats and CollectionStats Can include or exclude DBs to make execution faster
Parameters
includeDBs
table with list of string, that are used to find all databases that will be included (rest will be excluded). Default: nil - all tables will be included.
PARSE_URL(connectionString)
PARSE_URL: Parse MongoDB connection in the Standard Connection String format
Returns Lua table with the properties username, password, hosts, dbname, parameters.
Note 1: Will throw error if parsing failed
Note 2: Does not support the DNS Seedlist Connection Format
Usage
-- Examples:
local test1 = "mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017,cluster0-shard-00-01-osrtv.mongodb.net:27017,cluster0-shard-00-02-osrtv.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true&w=majority"
local test2 = "mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017/test"
local test3 = "mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017"
local test4_fail1 = "mongodb://myuser@cluster0-shard-00-00-osrtv.mongodb.net:27017"
local test5_fail2 = "mongodb://cluster0-shard-00-00-osrtv.mongodb.net"SET_STORES(self,connStr,authType,ignoreSystemLog,storeRootPath,noPasswordOverride)
Set the MongoDB connection properties for system object for custom datastores
Parameters
connectionString
Connection as connection string or or table with fields connection, username, password
authType
Specify authentication type as syslib.model.codes.MongoDBAuthentication (Default MONGAUTH_SCRAM_SHA_256).
ignoreSystemLog
If set to true the SystemLog system store will be ignored, and keep its old property values (Default nil).
Usage
-- Example 1: Set all stores to localhost MongoDB, with no authentication
return (require 'esi-mongo'):SET_STORES({
hosts={"localhost:27017"},
username="",
password=""
}, inmation.model.codes.MongoDBAuthentication.MONGAUTH_NONE)
-- Example 2: Set all stores to PSA replicaset, with default authentication
return (require 'esi-mongo'):SET_STORES({
hosts={"mongo1:27017", "mongo2:27017", "mongo3:27019"},
username="so",
password="inmation",
parameters = {replicaSet="rs0"}
})
-- Example 3: Set all stores to standalone MongoDB, with authentication
return (require 'esi-mongo'):SET_STORES("mongodb://myuser:<password>@cluster0-shard-00-00-osrtv.mongodb.net:27017")CREATE_CUSTOM_STORES(self,corePath,namePrefix,ignoreSets,retentionTimeDays1,retentionTimeDays2,addInfStore2,localDS_hasLocal,localDS_hasMaster,localDS_hasMasterLocal)
Create custom datastores according to standard naming convention
Naming convention: {CorePath}/Data Stores/{namePrefix}
Custom datastores for all 3 datatypes will be created (Timeseries, Events, Production Tracking). Each type will have 2 stores (purged / non-purged).
Datastore sets will be created for timeseries stores.
When executed on Local-Core, Datastore sets will be created for 3 variants: Local / Master / storage on both Local and Master-Core.
Parameters
retentionTimeDays1
Configure retention time for store 1 (non-Gxp). Default = 3650. If set to -1, purging is switched off.
Usage
local MONGO = require "esi-mongo"
local corePath = syslib.getcorepath()
local prefix = "GLOBAL"
local ignoreSets = false
local retentionTimeDays1 = 1 * 365 -- 1 year
local retentionTimeDays2 = 1 * 365 -- 1 year
local addInfStore2 = true
--local localDS_hasLocal = true
--local localDS_hasMaster = true
local localDS_hasMasterLocal = true
MONGO:CREATE_CUSTOM_STORES(corePath, prefix, ignoreSets, retentionTimeDays1, retentionTimeDays2, addInfStore2, localDS_hasLocal, localDS_hasMaster, localDS_hasMasterLocal)