I have a Loggly free trial account set up and receiving events . The JSON is an array of objects, devliered over a webhooks https POST. Each object is an event (as would seem reasonable for a bulk load interface). Simplified example:
[
{
"msys": {
"track_event": {
"event_id": "319115158633969504",
"friendly_from": "traffic.gen@example.com"
}
}
},
{
"msys": {
"track_event": {
"event_id": "319115158633970211",
"friendly_from": "traffic2.gen@example.com"
}
}
}
]
Can Loggly parse the contents of the JSON and extract values (e.g. event_id)? I've tried using the "Create Derived Fields" dialog, but this seems to be regex/line based rather than having JSON awareness.
Logically, I thought passing a JSON array would work, however after much trial-and-error, I found if you pass multiple, separate JSON objects separated by a newline character, these get parsed as separate events:
curl -H "content-type:application/json" -d $'{"timestamp":"2018-09-02T17:16:15.003123Z", "message":"test1"}\n{"timestamp":"2018-09-02T17:17:15.003123Z", "message":"test2"}' http://logs-01.loggly.com/bulk/<your-token>
A couple of things do bother me with this method (e.g. its not strictly, valid application/json you are passing, despite the content-type, and its not a JSON array), but this may get you over the hump in the problems you are having, but I'm hoping there is a more elegant answer than this.
See the following for more information: Loggly - Bulk Endpoint and Loggly - Automated Parsing