azureazure-sentinel

Need help to understand if azure sentinel data connection solution is being built correctly


My requirement is to develop and publish a solution. Workbooks, hunting queries, analytic rules, data connectors and more will be part of the solution.

Overall, customers who use this solution should be able to provide an AWS S3 bucket as input and allow this solution to ingest data from that bucket into custom tables defined in their log analytics workspace.

For the data connector part:

  1. It has to talk to AWS S3 buckets and ingest data into custom tables defined in log analytics workspace.
  2. Custom tables are built based on DCR.
  3. An Azure Function will be used to trigger a script
  4. Script is written in python that connects to the bucket that customer provides when they deploy this solution. Once connected, script reads data from the bucket and sends events in a batch over to sentinel using log ingestion api. Some instructions are here: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-api?source=recommendations

My question is, is this the right direction for building the data connector part of this solution.


Solution

  • You can use the below given code to send the custom logs using timer trigger function.

    import json
    import azure.functions as func
    from datetime import datetime
    import requests
    
    app = func.FunctionApp()
    
    @app.schedule(schedule="0 * * * * *", arg_name="myTimer", run_on_startup=True,
                  use_monitor=False) 
    def timer_trigger(myTimer: func.TimerRequest) -> None:
        
        time_generated = datetime.now().strftime('%Y-%m-%d %H:%M:%S') 
        logCombined = [
            {
            "TimeGenerated": time_generated,
            "Name": "Ikhtesam",
            "Computer": "Computer1",
            "AdditionalContext": "context-1"
          },
          {
            "TimeGenerated": time_generated,
            "Name": "Afreen",
            "Computer": "Computer2",
            "AdditionalContext": "context-2"
          }
        ]
        payload = json.dumps(logCombined)
        tenantId = "{tenantId}"
        clientId = "{clientId}"
        clientSecret = "{clientSecret}"
        scope = "https://monitor.azure.com/.default"
        dceUri = "https://******.eastus-1.ingest.monitor.azure.com"
        dcrImmutableId = "dcr-2e7e*******2de1"
        table = "DCR_Data_CL"
    
        body = f"client_id={clientId}&scope={scope}&client_secret={clientSecret}&grant_type=client_credentials"
        headers = {"Content-Type": "application/x-www-form-urlencoded"}
        uri = f"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token"
        response = requests.post(uri, data=body, headers=headers)
        bearerToken = response.json().get("access_token")
        
        headers2 = {"Authorization": f"Bearer {bearerToken}", "Content-Type": "application/json"}
        uri = f"{dceUri}/dataCollectionRules/{dcrImmutableId}/streams/Custom-{table}?api-version=2023-01-01"
        uploadResponse = requests.post(uri, data=payload, headers=headers2)
        print("Response: ", uploadResponse.status_code) 
    

    requirement.txt-

    azure-functions
    requests
    

    While executing, I am getting the expected output.

    enter image description here

    enter image description here