azureazure-data-factoryazure-data-exploreradx

Mapping JSON Object to ADX dynamic column in Azure Data Factory

I am trying to ingest a JSON file blob into an Azure Data Explorer table using the copy object in Azure Data Factory. To then transform and process the data afterward.

Is there a way to map the 'Properties' object to a dynamic column in ADX? I have tried creating an ingestion mapping in ADX and referencing that in the sink settings of the copy. I got an error that the mapping reference was not found.

If anyone has achieved this before, advice would be welcome. Thanks

ADF Mapping

Solution

  • You can ingest JSON data into an ADX table by using this.

    https://learn.microsoft.com/en-us/azure/data-explorer/ingest-json-formats?tabs=kusto-query-language#ingest-mapped-json-records

    You don't need to create a Copy pipeline, instead use Azure Data Explorer Command from ADF and run the following as a command

    .ingest into table Events ('https://kustosamplefiles.blob.core.windows.net/jsonsamplefiles/simple.json') with '{"format":"json", "ingestionMappingReference":"FlatEventMapping"}'

    I have ingested JSON which is stored in a blob container into a table in ADX using this approach.