I import data to my ELK stack using the Bulk API.
{"index":{"_index":"waf","_type":"logs","_id":"325d05bb6900440e"}}
{"id":"325d05bb6900440e","country":"US","ip":"1.1.1.1","protocol":"HTTP/1.1","method":"GET","host":"xxxxx","user_agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","uri":"/?a=><script>alert(1)</script>","request_duration":1999872,"triggered_rule_ids":["100030"],"action":"challenge","cloudflare_location":"unknown","occurred_at":"2017-01-23T17:38:58.46Z","rule_detail":[{"id":"","description":"ARGS:A"}],"rule_message":"Generic XSS Probing","type":"waf","rule_id":"100030"}
I have an ip in the data that i want to turn in to longitude and latitude using the GEOIP addon.
I have created a pipleine:
PUT _ingest/pipeline/geoip-info
{
"description": "Add geoip info",
"processors": [
{
"geoip": {
"field": "ip",
"target_field": "client_geoip",
"properties": ["location"],
"ignore_failure": true
}
}
]
}`
However when I import the data the pipeline is ignored can someone explain how i modify the bulk API to pass the information through a pipeline in order to add long and lat for me to create maps.
Thanks
In your bulk call you're missing the pipeline name
here
|
V
{"index":{"_index":"waf","_type":"logs","_id":"325d05bb6900440e", "pipeline": "geoip-info"}}
{"id":"325d05bb6900440e","country":"US","ip":"1.1.1.1","protocol":"HTTP/1.1","method":"GET","host":"xxxxx","user_agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36","uri":"/?a=><script>alert(1)</script>","request_duration":1999872,"triggered_rule_ids":["100030"],"action":"challenge","cloudflare_location":"unknown","occurred_at":"2017-01-23T17:38:58.46Z","rule_detail":[{"id":"","description":"ARGS:A"}],"rule_message":"Generic XSS Probing","type":"waf","rule_id":"100030"}
Or you can also set it in the bulk URL
POST _bulk?pipeline=geoip-info