amazon-web-servicesparsingaws-lambdalogparseraws-cloudwatch-log-insights

How to correctly parse lambda's whitespace separated log in Logs Insights


i have this type of log in @message:

2022-06-16T10:35:12.921Z 8984a0e4-0ff0-4cfd-ac5a-a312ec3f6157 DEBUG successfully retrieved 15758 object

How can i parse this log to have different columns with: timestamp (2022-06-16T10:35:12.921Z), requestID (8984a0e4-0ff0-4cfd-ac5a-a312ec3f6157), type(DEBUG), message (successfully retrieved 15758 object)?

I've tried with this query but the field type and message are not correctly recognized:

fields @timestamp, @message, @requestId
| parse @message "* * * *" as timestamp, requestId, type, message
| display @timestamp, @requestId, type, message 

the result is that timestamp and requestID are correctly identified, while in type there is "15758" instead of DEBUG and in message there is "object"....

how can i modify this query to have the correct output fields?


Solution

  • The parsing engine for CloudWatch Logs insights supports using regular expressions, so you can use a regex to obtain the desired result:

    fields @timestamp, @message
    | parse @message /^(?<timestamp>[^\s]+) (?<requestId>[^\s]+) (?<type>[^\s]+) (?<message>.+$)/
    | display timestamp, requestId, type, message
    

    will result in:

    enter image description here

    To see how this regex works in action you can use this regex101 link.