jsonwebhooksloggly

Can Loggly parse (derive values from) a JSON array (received using HTTP/S Bulk Endpoint)


I have a Loggly free trial account set up and receiving events Loggly screen. The JSON is an array of objects, devliered over a webhooks https POST. Each object is an event (as would seem reasonable for a bulk load interface). Simplified example:

[
  {
    "msys": {
      "track_event": {
        "event_id": "319115158633969504",
        "friendly_from": "traffic.gen@example.com"
      }
    }
  },
  {
    "msys": {
      "track_event": {
        "event_id": "319115158633970211",
        "friendly_from": "traffic2.gen@example.com"
      }
    }
  }
]

Can Loggly parse the contents of the JSON and extract values (e.g. event_id)? I've tried using the "Create Derived Fields" dialog, but this seems to be regex/line based rather than having JSON awareness.


Solution

  • Logically, I thought passing a JSON array would work, however after much trial-and-error, I found if you pass multiple, separate JSON objects separated by a newline character, these get parsed as separate events:

    curl -H "content-type:application/json" -d $'{"timestamp":"2018-09-02T17:16:15.003123Z", "message":"test1"}\n{"timestamp":"2018-09-02T17:17:15.003123Z", "message":"test2"}' http://logs-01.loggly.com/bulk/<your-token>
    

    A couple of things do bother me with this method (e.g. its not strictly, valid application/json you are passing, despite the content-type, and its not a JSON array), but this may get you over the hump in the problems you are having, but I'm hoping there is a more elegant answer than this.

    See the following for more information: Loggly - Bulk Endpoint and Loggly - Automated Parsing