azureterraformterraform-provider-azureazure-log-analytics

Can't create data collection rule for Azure log analytics using Terraform - missing Terraform options?


I'm creating a Terraform module to deploy an Azure Log Analytics workspace, a data collection endpoint and a data collection rule. Now the first two are working just fine, the last one is giving me issues.

The destination I'm using is my Log Analytics workspace, that seems to be fine. For both the Stream_declaration and data flows, I'm using one called "Custom-Logs_CL". The problem seems to be situated somewhere around that.

This is the error I'm getting:

│ Error: creating Data Collection Rule (Subscription: "subscriptionID"
│ Resource Group Name: "rg-phr-loganalytics-dev-weu"
│ Data Collection Rule Name: "pharos-cpi-rule"): datacollectionrules.DataCollectionRulesClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidPayload" Message="Data collection rule is invalid" Details=[{"code":"InvalidOutputTable","message":"Table for output stream 'Custom-Logs_CL' is not available for destination 'workspaceID'.","target":"properties.dataFlows[0]"},{"code":"InvalidOutputTable","message":"Custom table for stream 'Custom-Logs_CL' is not available for destination 'workspaceID'.","target":"properties.dataFlows[0]"}]
│ 
│   with module.log_analytics["pharos-cpi"].azurerm_monitor_data_collection_rule.rule,
│   on modules\log_analytics\main.tf line 16, in resource "azurerm_monitor_data_collection_rule" "rule":
│   16: resource "azurerm_monitor_data_collection_rule" "rule" {
│ 
│ creating Data Collection Rule (Subscription:
│ "Subscriptionid"
│ Resource Group Name: "rg-phr-loganalytics-dev-weu"
│ Data Collection Rule Name: "pharos-cpi-rule"):
│ datacollectionrules.DataCollectionRulesClient#Create: Failure responding to
│ request: StatusCode=400 -- Original Error: autorest/azure: Service returned
│ an error. Status=400 Code="InvalidPayload" Message="Data collection rule is
│ invalid" Details=[{"code":"InvalidOutputTable","message":"Table for output
│ stream 'Custom-Logs_CL' is not available for destination
│ 'e877bd70-2fca-429f-a7d4-183fbfe7a044'.","target":"properties.dataFlows[0]"},{"code":"InvalidOutputTable","message":"Custom
│ table for stream 'Custom-Logs_CL' is not available for destination
│ 'SubscriptionID'.","target":"properties.dataFlows[0]"}]

To me, it sounds that it's complaining about the Log Analytics workspace table that does not exist yet, so, I tried adding a Terraform block for this. I used this command, but as the documentation states; "This resource does not create or destroy tables. "

So a two-headed question, is this table not existing really my issue, and if so, is there another way to deploy it (using Terraform)?

This is the major part of my Terraform code:

resource "azurerm_monitor_data_collection_rule" "rule" {
  name                        = var.rule_name
  resource_group_name         = var.resource_group_name
  location                    = var.location
  tags                        = merge(var.tags, {})
  data_collection_endpoint_id = azurerm_monitor_data_collection_endpoint.endpoint.id

  destinations {
    log_analytics {
      workspace_resource_id = azurerm_log_analytics_workspace.log_analytics_workspace.id
      name                  = azurerm_log_analytics_workspace.log_analytics_workspace.workspace_id
    }

  }

  stream_declaration {
    stream_name = var.stream_declarations.name
    dynamic "column" {
      for_each = var.stream_declarations.column

      content {
        name = column.value.name
        type = column.value.type
      }
    }
  }

  dynamic "data_flow" {
    for_each = var.data_flows
    content {
      destinations       = [azurerm_log_analytics_workspace.log_analytics_workspace.workspace_id]
      streams            = data_flow.value.streams
      built_in_transform = data_flow.value.built_in_transform
      output_stream      = data_flow.value.output_stream
      transform_kql      = data_flow.value.transform_kql
    }
  }
}

Solution

  • As @vinay B confirmed, the table and rule need to be created manually, no option to do this using Terraform yet.