azure azure-resource-manager azure-data-explorer azure-eventhub azure-eventgrid

EventHubValidationErrorFound mapping does not exist

I got this error when I deployed the template and parameter below:

Error: Code: EventHubValidationErrorFound Message: [BadRequest] Validation Errors found: mapping does not exist

Basically what I am trying to do is to create a data connection Even Grid on a given Kusto cluster using ARM templates deployment.

I have this ARM template:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "kustoclusterName": {
      "type": "string"
    },
    "kustoDbName": {
      "type": "string"
    },
    "dataFormatType": {
      "type": "string",
      "defaultValue": "json"
    },
    "kustoDataConnectionsName": {
      "type": "string"
    },
    "subscriptionId": {
      "type": "string",
      "defaultValue": "[subscription().subscriptionId]"
    },
    "resourceGroup": {
      "type": "string",
      "defaultValue": "[resourceGroup().name]"
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    },
    "dpuMI": {
      "type": "string"
    },
    "kind": {
      "type": "string",
      "allowedValues": [
        "EventHub",
        "EventGrid"
      ]
    },
    "eventHubNamespace": {
      "type": "string"
    },
    "eventHub": {
      "type": "string"
    },
    "eventHubSubscriptionId": {
      "type": "string",
      "defaultValue": "[subscription().subscriptionId]"
    },
    "eventHubResourceGroupName": {
      "type": "string",
      "defaultValue": "[resourceGroup().name]"
    },
    "consumergroup": {
      "type": "string",
      "defaultValue": "$Default"
    },
    "storageAccountName": {
      "type": "string"
    },
    "mappingRuleName": {
      "type": "string"
    },
    "tableName": {
      "type": "string"
    },
    "dataFormat": {
      "type": "string",
      "allowedValues": [
        "JSON",
        "MULTIJSON",
        "CSV",
        "PARQUET"
      ]
    },
    "description": {
      "type": "string",
      "defaultValue": ""
    }
  },
  "variables": {},
  "resources": [
    {
      "condition": "[equals(parameters('kind'), 'EventHub')]",
      "type": "Microsoft.Kusto/clusters/databases/dataConnections",
      "apiVersion": "2021-08-27",
      "name": "[concat(parameters('kustoclusterName'), '/', parameters('kustoDbName'), '/', parameters('kustoDataConnectionsName'))]",
      "location": "[parameters('location')]",
      "kind": "[parameters('kind')]",
      "properties": {
        "managedIdentityResourceId": "[parameters('dpuMI')]",
        "eventHubResourceId": "[resourceId(parameters('eventHubSubscriptionId'), parameters('eventHubResourceGroupName'), 'Microsoft.EventHub/namespaces/eventhubs', parameters('eventHubNamespace'), parameters('eventHub'))]",
        "consumerGroup": "[parameters('consumergroup')]"
      },
      "tags": {
        "Version": "0.0.0.0",
        "Description": "[parameters('description')]"
      }
    },
    {
      "condition": "[equals(parameters('kind'), 'EventGrid')]",
      "type": "Microsoft.Kusto/clusters/databases/dataConnections",
      "apiVersion": "2021-08-27",
      "name": "[concat(parameters('kustoclusterName'), '/', parameters('kustoDbName'), '/eg', parameters('kustoDataConnectionsName'))]",
      "location": "[parameters('location')]",
      "kind": "[parameters('kind')]",
      "properties": {
        "ignoreFirstRecord": "[equals(parameters('dataFormat'), 'CSV')]",
        "storageAccountResourceId": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]",
        "managedIdentityResourceId": "[parameters('dpuMI')]",
        "eventHubResourceId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroup'), 'Microsoft.EventHub/namespaces/eventhubs', parameters('eventHubNamespace'), parameters('eventHub'))]",
        "consumerGroup": "[parameters('consumergroup')]",
        "tableName": "[parameters('tableName')]",
        "dataFormat": "[parameters('dataFormat')]",
        "blobStorageEventType": "Microsoft.Storage.BlobCreated",
        "mappingRuleName": "[parameters('mappingRuleName')]"
      },
      "tags": {
        "Version": "0.0.0.0",
        "Description": "[parameters('description')]"
      }
    }
  ]
}

Parameter:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "0.0.0.0",
  "parameters": {
    "eventHubNamespace": {
      "value": "cpcnx-ehns-adlsv2tokusto-test01"
    },
    "eventHub": {
      "value": "cpcnx-eh-cds-storeingest-adxingest-dbingest-test01"
    },
    "consumergroup": {
      "value": "dpucg"
    },
    "kustoclusterName": {
      "value": "cpcnx-adx-test01"
    },
    "kustoDbName": {
      "value": "ingest"
    },
    "kustoDataConnectionsName": {
      "value": "cdsstoreingestadxingestdbingest"
    },
    "eventHubSubscriptionId": {
      "value": "{subId}"
    },
    "eventHubResourceGroupName": {
      "value": "cpcnx-test-test01"
    },
    "dpuMI": {
      "value": "/subscriptions/{subId}/resourceGroups/{regGroup}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cpcnx-pipmi-test01"
    },
    "kind": {
      "value": "EventGrid"
    },
    "storageAccountName": {
      "value": "cpcnxsaingesttest01"
    },
    "mappingRuleName": {
      "value": "EHCPCDevices_FastStream_Mapping"
    },
    "tableName": {
      "value": "EHCPCDevices_FastStream"
    },
    "dataFormat": {
      "value": "PARQUET"
    }
  }
}

Solution

Validation Errors found: mapping does not exist:

Check below steps to resolve the error "Mapping does not exist":

Make sure that the mapping rule existed in the specific table you are using in the ARM template.
Check if that the mapping rule has been created successfully in the Kusto database and existed in the specific Kusto cluster.
Also verify the cluster, database and data table names are correctly given according to their existence without any conflict.

Run below query in the Kusto explorer query editor to verify that the mapping rules actually existed or not in the Kusto database.

.show table lattable ingestion parquet mappings

enter image description here

If nothing from the above resolves the issue, create a new ingestion mapping rule with the required data format by referring to the mapping formats available in Kusto explorer and redeploy the template with the modified parameter names.

Your code looks good for the requirement. Referring to this MSDoc ARM template, I have created a data connection between Azure event grid and a data explorer. It was successfully deployed as shown.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "eventhubnamespace": {
            "type": "string",
            "defaultValue": "cpcnx-ehns-adlsv2tokusto-test0",
            "metadata": {
                "description": "event hub namespace name."
            }
        },
        "eventhubname": {
            "type": "string",
            "defaultValue": "cpcnx-eh-cds-storeingest-adxingest-dbingest-test01",
            "metadata": {
                "description": "event hub name."
            }
        },
        "consumergroup_default_name": {
            "type": "string",
            "defaultValue": "$Default",
            "metadata": {
                "description": "consumer group of the event hub."
            }
        },
        "storageaccount": {
            "type": "string",
            "defaultValue": "caronewperfdiag162",
            "metadata": {
                "description": "storage account name"
            }
        },
        "kustocluster": {
            "type": "string",
            "defaultValue": "cpcnx-adx-test0"
        },
        "kustodb": {
            "type": "string",
            "defaultValue": "ingest",
            "metadata": {
                "description": "database"
            }
        },
        "kustotable": {
            "type": "string",
            "defaultValue": "lattable",
            "metadata": {
                "description": "table"
            }
        },
        "mappingRuleName": {
          "type": "string",
          "defaultValue":"Mapping1"
       },
        "dataformat_type": {
            "type": "string",
            "defaultValue": "PARQUET",
            "metadata": {
                "description": "data format"
            }
        },
             "databaseRouting_type": {
            "type": "string",
            "defaultValue": "Single"
        },
        "dataconnections_kustodc_name": {
            "type": "string",
            "defaultValue": "kustodcnew",
            "metadata": {
                "description": "data connection to create"
            }
        },
        "subscriptionId": {
            "type": "string",
            "defaultValue": "[subscription().subscriptionId]",
            "metadata": {
                "description": "subscriptionId of the resources"
            }
        },
        "resourceGroup": {
            "type": "string",
            "defaultValue": "[resourceGroup().name]",
            "metadata": {
                "description": "resourceGroup of the resources"
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "[resourceGroup().location]",
            "metadata": {
                "description": "Location"
            }
        }
    },
    "variables": {
    },
    "resources": [{
            "type": "Microsoft.Kusto/Clusters/Databases/DataConnections",
            "apiVersion": "2022-02-01",
            "name": "[concat(parameters('kustocluster'), '/', parameters('kustodb'), '/', parameters('dataconnections_kustodc_name'))]",
            "location": "[parameters('location')]",
            "kind": "EventGrid",
            "properties": {
                "managedIdentityResourceId": "[resourceId('Microsoft.Kusto/clusters', parameters('kustocluster'))]",
                "storageAccountResourceId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroup'), 'Microsoft.Storage/storageAccounts', parameters('storageaccount'))]",
                "eventHubResourceId": "[resourceId(parameters('subscriptionId'), parameters('resourceGroup'), 'Microsoft.EventHub/namespaces/eventhubs', parameters('eventhubnamespace'), parameters('eventhubname'))]",
                "consumerGroup": "[parameters('consumergroup_default_name')]",
                "tableName": "[parameters('kustotable')]",
                "dataFormat": "[parameters('dataformat_type')]",
                "mappingRuleName": "[parameters('mappingRuleName')]",
                "databaseRouting": "[parameters('databaseRouting_type')]"
            }
        }
    ]
}

Output:

enter image description here