azure-data-explorerkqldata-ingestion

Azure Data Explorer ingest text Log Files with custom delimiter


I'm trying to use Azure Data Explorer to ingest some logs (IIS Logs, POP3 logs, IMAP logs) that contain values delimited by space.

I would have expected Azure Data Explorer to infer the correct schema from the files as separate columns, however it only identifies a single column with the entire data.

enter image description here

The reason for this seems to be the header and metadata rows, which I can't find a way to skip (would have thought there is a way to skip those).

However, even if I remove the metadata rows, manually, from the log file, it still doesn't seem to be able to recognize the schema for the table.

enter image description here

I have also tried to create the table before ingesting, using KQL queries, and instead of creating a new table, I ask the import to ingest into an already existing table. However, doing this, it doesn't identify any rows to be imported from the logs.

enter image description here

I'm not sure what exactly can be done, I thought Azure Data Explorer (and Log Explorer - tried that too, works the same) to be a perfect solution for log files created by Windows apps.


Solution

  • The documentation might have been a good start point.
    It is very clear as to what are the supported formats for ingestion.
    IIS Logs, POP3 logs & IMAP logs are not listed.
    Data formats supported by Azure Data Explorer for ingestion

    As to the TXT format, an entire line is ingested as a single value. No additional parsing there.

    Format Extension Description
    TXT .txt A text file with lines delimited by \n. Empty lines are skipped.

    You could use the TXT format to load the data and then parse it and split it to columns, within ADX, probably by using REGEX.