sumologic

Extracting json value from json within a log entry

Using sumologic, how can I perform queries on log entries where the message is partially json (so not entirely)?

Example entry:

[2020-04-30 02:10:32] production.WARNING: We were rate limited {"class":"App\WebhookService\WebhookExecutor","headers":{"Date":["Thu, 30 Apr 2020 02:10:32 GMT"],"Content-Type":["application/json"],"Content-Length":["88"],"Connection":["keep-alive"],"Set-Cookie":...

I have a lot of these kinds of errors with different class paths, some are webhooks executions, others are different objects. My end goal is to be able to count the log entries for each class name, like this:

WebhookExecutor     | 156
SyncDataService     | 48
DeleteService       | 13

I'm starting with something like:

"We were rate limited" | json auto | json "class"

My expectation with ^ was that it would create a column called class in my Log Search results that I could then group by, count, etc.

Solution

  • Not sure what do you mean by "partially Json"? I.e.:

    1. that the log message contains other text and a proper Json
    2. or that the message contains something which is similar to Json, but not really a Json - e.g. a prefix of a Json message

    Assuming Option 1 (other text + proper Json)

    You need to extract your Json to a proper field of its own. Otherwise | json auto is not able to pick it up.

    So:

    "We were rate limited"
| parse "We were rate limited *" as limitedJson
| json auto
| count by class

    should do the job. The first line is not even needed, because parse does the filtering too.

    Assuming Option 2 (no proper Json, just something similar)

    It's harder in such a case, but still doable.

    I haven't tested it, but something like this should work:

    "We were rate limited"
| parse "\"class\":\"*\"" as class
| count by class

    Here escaping the nested quotes doesn't look well.

    Disclaimer: I am currently employed by Sumo Logic.