.netazurepowershellkqlazure-data-explorer

How to ingest inline into Azure Data Explorer Table from PowerShell?


Inline ingestion into an Azure Data Explorer Table is possible through the .ingest inline control command:

.ingest inline into table MyTable with (format = 'csv') <|
    "this","inline","CSV","gets","ingested","to","MyTable"

This self-answer provides guidance on how to ingest inline from PowerShell using the Kusto.Data assembly from Kusto.Tools package.

Usage of the assembly is partially documented in Use Kusto .NET client libraries from PowerShell, however it does not cover inline ingestion.


Solution

  • An alternative to my other answer to execute control commands can be using Invoke-KustoControlCommand from the PowerShellKusto Module. The module is an abstraction over the Microsoft.Azure.Kusto.Data and Microsoft.Azure.Kusto.Ingest assemblies to simplify Azure Data Explorer management, ingestion and querying.

    # NOTE: This module requires PowerShell 7.2+
    Install-Module PowerShellKusto -Scope CurrentUser
    
    # This method prompts for authentication,
    # the cmdlet also offers other authentication methods:
    #
    # - System or User Managed Identity
    # - Certificate via X509Certificate2 or Thumbprint
    # - Secret
    # - User and Application Access Token
    #
    # See the docs for more details.
    
    $connectKustoSplat = @{
        Database = 'myDb'
        Cluster  = 'https://mySourceCluster.eastus.kusto.windows.net'
    }
    Connect-Kusto @connectKustoSplat
    
    $path = 'path\to\csvToIngest.csv'
    $tableName = 'myTable'
    # set to false if the CSV has no headers
    $ignoreFirstRecord = $true
    
    $reqProps = New-KustoClientRequestProperties -ServerTimeout '00:00:30'
    Invoke-KustoControlCommand -RequestProperties $reqProps "
    .ingest inline into table $tableName with (
        format = 'csv',
        ignoreFirstRecord = $ignoreFirstRecord) <|
    $([System.IO.File]::ReadAllText($path))"
    

    Above example shows inline ingestion, however a much easier way to ingest the file in this case is with Invoke-KustoIngestFromStorage:

    Install-Module PowerShellKusto -Scope CurrentUser
    
    $connectKustoSplat = @{
        Database = 'myDb'
        Cluster  = 'https://mySourceCluster.eastus.kusto.windows.net'
    }
    Connect-Kusto @connectKustoSplat
    
    $tableName = 'myTable'
    
    $invokeKustoIngestFromStorageSplat = @{
        Table             = $tableName
        Format            = 'csv'
        IgnoreFirstRecord = $true
        Path              = 'path\to\csvToIngest.csv'
    }
    Invoke-KustoIngestFromStorage @invokeKustoIngestFromStorageSplat