Connect VM to Azure Log Analytics Workspace (Sentinel) with Data Collection Rule (Terraform)

I'm working on setting Microsoft Sentinel connect to my VM for my personal projects using Terraform. I have successfully configured my Log Analytics Workspace, Sentinel, and VM Extension. However, when I attempt to create the Data Collection Rule (DCR), I keep receiving a "400 error" indicating an "InvalidPayload.":

Error: creating Data Collection Rule (Subscription: "28a9287a-502c-4b96-aea4-34779d8ca1b4"
│ Resource Group Name: "sentinel-lab"
│ Data Collection Rule Name: "sentinel-dcr"): unexpected status 400 (400 Bad Request) with error: InvalidPayload: Data collection rule is invalid  
│
│   with azurerm_monitor_data_collection_rule.sentinel-dcr,
│   on sentinel.tf line 33, in resource "azurerm_monitor_data_collection_rule" "sentinel-dcr":
│   33: resource "azurerm_monitor_data_collection_rule" "sentinel-dcr" {
│
│ creating Data Collection Rule (Subscription: "28a9287a-502c-4b96-aea4-34779d8ca1b4"
│ Resource Group Name: "sentinel-lab"
│ Data Collection Rule Name: "sentinel-dcr"): unexpected status 400 (400 Bad Request) with error: InvalidPayload: Data collection rule is invalid 

Below is my code for the Log Analytics Workspace, Sentinel, DCR, and VM Extension using Terraform. All services are configured correctly except for the DCR part, I think. I've been modifying the DCR code for three days and even added a Sentinel onboarding block, but it produced another error, so I believe it doesn't matter.

#Create Log Analytics Workspace

resource "azurerm_log_analytics_workspace" "sentinel-log" {

  name                = "sentinel-log"

  location            = azurerm_resource_group.sentinel-lab.location

  resource_group_name = azurerm_resource_group.sentinel-lab.name

  sku                 = "PerGB2018"

  retention_in_days   = 30

}





# Enable Microsoft Sentinel by adding the SecurityInsights solution

resource "azurerm_log_analytics_solution" "sentinel" {

  solution_name         = "SecurityInsights"

  location              = azurerm_resource_group.sentinel-lab.location

  resource_group_name   = azurerm_resource_group.sentinel-lab.name

  workspace_name        = azurerm_log_analytics_workspace.sentinel-log.name`



  plan {

    publisher = "Microsoft"

    product   = "OMSGallery/SecurityInsights"

  }



  workspace_resource_id = azurerm_log_analytics_workspace.sentinel-log.id

}



Data Collection Rule on AMA

resource "azurerm_monitor_data_collection_rule" "sentinel-dcr" {

  name                = "sentinel-dcr"

  location            = azurerm_resource_group.sentinel-lab.location

  resource_group_name = azurerm_resource_group.sentinel-lab.name`



  data_sources {

    windows_event_log {

      name = "Windows-Log"

      x_path_queries = ["*![System/Level=1]"]  # Optional filtering

      streams        = ["Application", "System", "Security"]

    }

  }



  destinations {

    log_analytics {

      name                  = "log-analytics"

      workspace_resource_id = azurerm_log_analytics_workspace.sentinel-log.id

    }

  }



  data_flow {

    streams      = ["Microsoft-InsightsMetrics"]

    destinations = ["log-analytics"]

  }

}



#Associate DCR with the Windows VM

resource "azurerm_monitor_data_collection_rule_association" "sentinel-dcr-association" {

  name                    = "sentinel-dcr-association"

  target_resource_id      = azurerm_windows_virtual_machine.sentinel-vm.id

  data_collection_rule_id = azurerm_monitor_data_collection_rule.sentinel-dcr.id

}



Extension

resource "azurerm_virtual_machine_extension" "sentinel-ama" {

  name                       = "AzureMonitorWindowsAgent"

  virtual_machine_id         = azurerm_windows_virtual_machine.sentinel-vm.id

  publisher                  = "Microsoft.Azure.Monitor"

  type                       = "AzureMonitorWindowsAgent"

  type_handler_version       = "1.0"

  auto_upgrade_minor_version = true

}

I have tried a lot of code, look through all the terraform documentation still can't find out a solution, can anyone help?

Connect VM to Azure Log Analytics Workspace Sentinel with Data Collection Rule using terraform

As per the issue, I can see two changes need to be done in the configuration. As per the official documentation from HashiCorp streams should be defined as mentioned below.

streams  =  ["Microsoft-SecurityEvent"]

But the one you used in the configuration doesn't match with the requirement.

And as per the Q&A Doc answered by AnuragSingh-MSFT when you're trying to get the event logs for filtering Level 1 and Level 2 security logs should be as mentioned below

x_path_queries = ["Security!*[System[(Level=1 or Level=2)]]"]

Demo configuration:

resource "azurerm_log_analytics_solution" "sentinel" {
  solution_name       = "SecurityInsights"
  location           = azurerm_resource_group.sentinel_lab.location
  resource_group_name = azurerm_resource_group.sentinel_lab.name
  workspace_name     = azurerm_log_analytics_workspace.sentinel_log.name

  plan {
    publisher = "Microsoft"
    product   = "OMSGallery/SecurityInsights"
  }

  workspace_resource_id = azurerm_log_analytics_workspace.sentinel_log.id
}


resource "azurerm_monitor_data_collection_rule" "sentinel_dcr" {
  name                = var.sentinel_dcr_name
  location            = azurerm_resource_group.sentinel_lab.location
  resource_group_name = azurerm_resource_group.sentinel_lab.name
  kind                = "Windows"
 
  data_sources {
    windows_event_log {
      name           = "Windows-Event-Log"
      x_path_queries = ["Security!*[System[(Level=1 or Level=2)]]"]  
      streams        = ["Microsoft-SecurityEvent"] 
    }
  }
  
  destinations {
    log_analytics {
      name                  = "log-analytics"
      workspace_resource_id = azurerm_log_analytics_workspace.sentinel_log.id
    }
  }
 
  data_flow {
    streams      = ["Microsoft-SecurityEvent"]  
    destinations = ["log-analytics"]
  }
}

resource "azurerm_monitor_data_collection_rule_association" "sentinel_dcr_association" {
  name                    = "sentinel-dcr-association"
  target_resource_id      = azurerm_windows_virtual_machine.sentinel_vm.id
  data_collection_rule_id = azurerm_monitor_data_collection_rule.sentinel_dcr.id
}

Deployment:

Once this setup is done, check with the log analytics workspace and kql query for logs

SecurityEvent
| where TimeGenerated > ago(30m) 
| order by TimeGenerated desc

Refer:

https://learn.microsoft.com/en-us/azure/azure-monitor/vm/data-collection?tabs=cli#filter-events-using-xpath-queries