pythonazuresyslogazure-log-analytics

Azure Log Analytics: How to send custom python pandas DataFrames into LAW

I have a Python script that I want to run daily that produces a Pandas dataframe.

I want this dataframe to be added daily to my log analytics workspace.

I have a Windows server that I can use to run my Python script.

What do I need to do to make this work? Is there a way to go from DataFrames to push to a Syslog server?

Solution

  • Azure Log Analytics: How to send custom python pandas DataFrames into LAW

    I have reproduced in my environment and below are my expected results:

    I have taken below code from Microsoft-Document and modified a little:

    import json
import requests
import datetime
import hashlib
import hmac
import base64

logtype2="RithwikLogs1"
def build_signature(customer_id, shared_key, date, content_length, method, content_type, resource):
    x_headers = 'x-ms-date:' + date
    string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
    bytes_to_hash = bytes(string_to_hash, encoding="utf-8")  
    decoded_key = base64.b64decode(shared_key)
    encoded_hash = base64.b64encode(hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()).decode()
    authorization = "SharedKey {}:{}".format(customer_id,encoded_hash)
    return authorization

def post_data1(customer_id, shared_key, body, log_type):
    method = 'POST'
    content_type = 'application/json'
    resource = '/api/logs'
    rfc1123date = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    content_length = len(body)
    signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)
    uri = 'https://' + customer_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'

    headers = {
        'content-type': content_type,
        'Authorization': signature,
        'Log-Type': log_type,
        'x-ms-date': rfc1123date
    }

    response = requests.post(uri,data=body, headers=headers)
    if (response.status_code >= 200 and response.status_code <= 299):
        print('Rithwik Data Frame is sent to Log Analytics Worksapce ')
    else:
        print("Response code: {}".format(response.status_code))
rithwik_data1 = pd.DataFrame({
    'Name': ['Rithwik', 'Bojja', 'Chotu'],
    'Age': [23, 23, 20]
})

post_data1('310603f7', 'a3m2jEErIT6HONEdrAhgIGBrT9L78AK0wk0H8HJKkEdTva4nmw==',rithwik_data1.to_json(orient="records")
, logtype2)

    Output:

    enter image description here

    enter image description here

    enter image description here

    After running the python code, it takes some time for the logs to be generated in Log Analytics(Wait for around 5-10 min).

    Note:

    In calling of function:

    post_data1('yourworkspaceid', 'Primarykey(sharedkey)',body(json of df), logtype2)

    You can get above values by following below:

    enter image description here