kqlazure-cloud-servicesazure-monitoringazure-log-analytics-workspace

KQL Query to filter duplicate entries and select top 1 from the log data


I am using below query to get the container error log and filtering to remove duplicates.

    let ContainerIdList = KubePodInventory
    | where ContainerName contains "acc-c1-logger"
    | where Namespace has "prd" | where ClusterId =~ '/subscriptions/xxxx/resourcegroups/xxxx/providers/Microsoft.ContainerService/managedClusters/aksprd'
   | distinct ContainerID;
   ContainerLog 
   | where ContainerID in (ContainerIdList)
   | where LogEntry !has "SRV1174"
   | where LogEntry has "| E |" or LogEntry has "| F |"
   | where LogEntry !contains "the I/O interface definition of project" 
   | where LogEntry !contains "the I/O interface definition of cuc" 
   | where TimeGenerated > ago(5m)
   | project LogEntrySource, LogEntry, TimeGenerated 
   | order by TimeGenerated desc
   | top 1000 by LogEntry
   | render table
   | extend SplitLog = split(LogEntry, "|")
   | project C1 = SplitLog[0], cc=SplitLog[1],C2 = todatetime(SplitLog[1]), C3 = 
     SplitLog[2], C4 = SplitLog[3], C5=SplitLog[4], logerror=SplitLog[5]
   | summarize arg_max(C2,*) by tostring(logerror)
   | project-away logerror
   | project ERROR = strcat( cc, "|", C3, "|", C4, "|", C5, "|", logerror1)

i get following output in the error table

  2024-06-27 20:43:47 | con-prc-sc | SRV2006 | E | [DB_AdviceSimulationAlerted] on project 
  'Advice': error while storing: During executeUpdate: Could not find prepared statement 
   with handle 7."
   2024-06-27 20:44:00 | con-prc-sc | SRV2001 | E | Unable to connect DB

   2024-06-27 20:44:47 | con-prc-sc | SRV2006 | E | [DB_AdviceSimulationAlerted] on project 
  'Advice': error while storing: During executeUpdate: Could not find prepared statement 
   with handle 9."
   2024-06-27 20:45:00 | con-prc-sc | SRV2001 | E | file is missing on relative path

   2024-06-27 20:45:47 | con-prc-sc | SRV2006 | E | [DB_Advice] on project 
  'Advice': error while storing: During executeUpdate: Could not find prepared statement 
   with handle 11."

the requirement is here , i need to get only top 1 from the could not fine prepared statement error along with other errors.

 The Expected outcome should be 
 Error:
 2024-06-27 20:43:47 | con-prc-sc | SRV2006 | E | 
 [DB_AdviceSimulationAlerted] on project 
 'Advice': error while storing: During executeUpdate: Could not find 
 prepared statement with handle 7."
 2024-06-27 20:44:00 | con-prc-sc | SRV2001 | E | Unable to connect DB
 2024-06-27 20:45:00 | con-prc-sc | SRV2001 | E | file is missing on 
 relative path

thanks in advance


Solution

  • You could do the following directly below your last line of your Query: Data is only for reproducing your data.

    let Data = datatable(ERROR: string)
    [
       "2024-06-27 20:43:47 | con-prc-sc | SRV2005 | E | [DB_AdviceSimulationAlerted] on project 'Advice': error while storing: During executeUpdate: Could not find prepared statement with handle 11.",
       "2024-06-27 20:43:47 | con-prc-sc | SRV2005 | E | [DB_AdviceSimulationAlerted] on project 'Advice': error while storing: During executeUpdate: Could not find prepared statement with handle 9.",
       "2024-06-27 20:43:47 | con-prc-sc | SRV2006 | E | [DB_AdviceSimulationAlerted] on project 'Advice': error while storing: During executeUpdate: Could not find prepared statement with handle 7.",
       "2024-06-27 20:44:00 | con-prc-sc | SRV2001 | E | Unable to connect DB",
       "2024-06-27 20:44:47 | con-prc-sc | SRV2006 | E | [DB_AdviceSimulationAlerted] on project 'Advice': error while storing: During executeUpdate: Could not find prepared statement with handle 9.",
       "2024-06-27 20:45:00 | con-prc-sc | SRV2001 | E | file is missing on relative path",
       "2024-06-27 20:45:47 | con-prc-sc | SRV2006 | E | [DB_Advice] on project 'Advice': error while storing: During executeUpdate: Could not find prepared statement with handle 11."
    ];
    Data
    | extend type = tostring(split(ERROR, "|")[2])
    | extend ts = tostring(split(ERROR, "|")[0])
    | extend message = tostring(split(ERROR, "|")[4])
    | extend type = iff(message contains "Could not find prepared statement with handle", "A", strcat(type, message))
    | order by type desc, ts desc
    | extend HasPrev = prev(type) == type
    | where HasPrev == false
    | project ERROR
    

    Demo here

    Result: enter image description here