Azure Machine Learning Studio Using Python

Azure Machine Learning Studio supports Python script execution, enabling the user to read and write data in system:inmation. Scripts can be compressed as zip files and zip files can be imported and saved as datasets in the Azure ML Studio.

The example script below contains a few basic functions for accessing data stored in the system. The script should be saved as a python file(*.py) and then added to a zip archive. The zip file can then be uploaded from the local computer as data set to Azure ML Studio.

Make a note of the saved filename as you will need it to reference in subsequent scripts that access the functions. For the following examples we use the name pythonclientexample.
import json
import requests
import datetime
import pandas as pd

class Client:
    baseURL = "http://localhost:8002"
    options = {
        "auth": {
            "username": '',
            "password": '',
            "authority": 'inmation',
            "grant_type": "password"
        }
    }

    def __init__(self, baseURL, options):
        self.baseURL = baseURL
        self.options = options

    def reqHeaders(self):
        headers = {
            "Content-Type": 'application/json'
        }
        options = self.options or {}
        auth = options["auth"] or {}
        headers["username"] = auth["username"]
        headers["password"] = auth["password"]
        headers["authority"] = auth["authority"]
        headers["grant_type"] = auth["grant_type"]
        return headers

    def createHistoryRequestItem(self, path, aggregate = None):
        agg = aggregate or 'AGG_TYPE_INTERPOLATIVE'
        return {
            "p": path,
            "aggregate": agg
        }

    def createItemValue(self, p, v, q, t):
        return {
            "p": p,
            "v": v,
            "q": q,
            "t": t
        }

    def readHistoricalData(self, items, start_time, end_time, intervals_no):
        reqBody = {
            "start_time": start_time,
            "end_time": end_time,
            "intervals_no": intervals_no,
            "items": items
        }
        headers = self.reqHeaders()
        url = "{}/api/v2/readhistoricaldata".format(self.baseURL)
        print("inmation Web API URL: '{}'".format(url))
        print("inmation Web API headers: '{}'".format(headers))
        print("inmation Web API body: '{}'".format(json.dumps(reqBody)))
        r = requests.post(url, data = json.dumps(reqBody), headers = headers)
        return r

    def write(self, itemvalues):
        reqBody = {
            "items": itemvalues
        }
        headers = self.reqHeaders()
        url = "{}/api/v2/write".format(self.baseURL)
        print("inmation Web API URL: '{}'".format(url))
        print("inmation Web API headers: '{}'".format(headers))
        print("inmation Web API body: '{}'".format(json.dumps(reqBody)))
        r = requests.post(url, data = json.dumps(reqBody), headers = headers)
        return r

Invoking function

Functions from the Zip file can be invoked by adding Execute Python Script tile. Example code for invoking the ReadHistoricalData endpoint:

import pandas as pd
from inmationhttpclient import Client

baseURL = "http://localhost:8002"
options = {
    "auth": {
        "username": '%USERNAME%',
        "password": '%PASSWORD%',
        "authority": 'inmation',
        "grant_type": "password"
    }
}

client = Client(baseURL, options)

def azureml_main(dataframe1 = None, dataframe2 = None):
    start_time = "2018-01-01T00:00:00.000Z"
    end_time = "2018-01-02T00:00:00.000Z"
    intervals_no = 31
    items = [
        {
            "p": "%PATH%",
            "aggregate": "AGG_TYPE_INTERPOLATIVE"
        },
        {
            "p": "%PATH%",
            "aggregate": "AGG_TYPE_INTERPOLATIVE"
        }
    ]
    r = client.readHistoricalData(items, start_time, end_time, intervals_no)

    print("JSON")
    jsonRes = r.json()
    print(type(jsonRes))
    dataFrameData = {}
    if isinstance(jsonRes, dict):
        print("r.json is an dictionary!!")
        data = jsonRes['data']
        if isinstance(data, dict):
            print("data is an dictionary!!")
            items = data['items']
            if isinstance(items, list):
                print("items is a list!!")
                for item in items:
                    objPath = item['p']
                    itempath = objPath.split('/')
                    objName = itempath[-1]
                    intervals = item['intervals']
                    v = []
                    q = []
                    t = []
                    print(type(v))
                    dataFrameData[objName + '_v'] = v
                    dataFrameData[objName + '_q'] = q
                    dataFrameData[objName + '_t'] = t
                    print(type(v))
                    if isinstance(intervals, list):
                        print("intervals is a list!!")
                        for interval in intervals:
                            v.append(interval['V'])
                            q.append(interval['Q'])
                            t.append(interval['T'])

    print(v, q, t)
    dataframe = pd.DataFrame(data = dataFrameData)
    return dataframe

The following arguments should be provided: item paths, start and end time, intervals and aggregation.

The zip file, which stores the inmation http Python client creation script should be connected to the Python script tile. After these files have been connected click run on the Python script tile.

Access input data

After successful execution, a green tick mark will be displayed at the top right corner of the screen.

To view the imported data, click the right mouse button, choose Result Data set, Visualize option. Various statistics about each column are displayed. Data can be modified and filtered, missing values deleted or replaced.

Store model data to system:inmation

After incoming data has been shaped and the machine learning module created, the output can be written to system:inmation.

Deploy another script execution tile in ML studio, connect the tile with data output and the zip fil. Create a new item in the I/O model, copy the full property path, fill it in the example script item path place holder and click on run.

After successful run, values are set to the item in the DataStudio.

import pandas as pd
from pythonclientexample import Client
import json

baseURL = "http://localhost:8002"
options = {
    "auth": {
        "username": '%USERNAME%',
        "password": '%PASSWORD%',
        "authority": 'inmation',
        "grant_type": "password"
    }
}

client = Client(baseURL, options)
p = "%PATH%"

def azureml_main(dataframe1 = None, dataframe2 = None):
    colnames = dataframe1.columns
    scores = dataframe1[colnames[6]]
    timestamps = dataframe1[colnames[0]]
    itemValues = []
    for index, timestamp in timestamps.iteritems():
        v = int(scores[index])
        t = int(timestamp)
        itemValues.append(
            {
                "p": p,
                "v": v,
                "t": t
            }
        )

    print(itemValues)
    r = client.write(itemValues)
    print(r.json())
    return dataframe1

What can your company create using Azure ML Studio and system:inmation?

We hope that the ease of data flow between system:inmation and the Azure ML Studio enables our customers to work freely with machine learning. Create your experiments and do not forget to share your successes with us.