I'm trying to use Azure Data Explorer to ingest some logs (IIS Logs, POP3 logs, IMAP logs) that contain values delimited by space.
I would have expected Azure Data Explorer to infer the correct schema from the files as separate columns, however it only identifies a single column with the entire data.
The reason for this seems to be the header and metadata rows, which I can't find a way to skip (would have thought there is a way to skip those).
However, even if I remove the metadata rows, manually, from the log file, it still doesn't seem to be able to recognize the schema for the table.
I have also tried to create the table before ingesting, using KQL queries, and instead of creating a new table, I ask the import to ingest into an already existing table. However, doing this, it doesn't identify any rows to be imported from the logs.
I'm not sure what exactly can be done, I thought Azure Data Explorer (and Log Explorer - tried that too, works the same) to be a perfect solution for log files created by Windows apps.
The documentation might have been a good start point.
It is very clear as to what are the supported formats for ingestion.
IIS Logs, POP3 logs & IMAP logs are not listed.
Data formats supported by Azure Data Explorer for ingestion
As to the TXT format, an entire line is ingested as a single value. No additional parsing there.
Format | Extension | Description |
---|---|---|
TXT | .txt | A text file with lines delimited by \n. Empty lines are skipped. |
You could use the TXT format to load the data and then parse it and split it to columns, within ADX, probably by using REGEX.