logstashlogstash-configurationelk

Logstash S3 output prefix - event date field value

How to set Logstash S3 output prefix dynamically with an event field value in format: "%{+YYYY}/%{+MM}/%{+dd}/%{+HH}" ?

input:
{"record_time":"2017-03-09T04:07:51.520Z"}

required s3 prefix: 
2017/03/09/04
Solution

  • You can use grok to match record_time to extract year, month, day, hour and then mutate into s3 prefix:

    grok {
  match => {
    "record_time" => "%{INT:year}-%{INT:month}-%{INT:day}T%{INT:hour}:%{GREEDYDATA}"
  }
}

mutate {
  # Create s3 prefix
  add_field => {
    "s3_prefix" => "%{year}/%{month}/%{day}/%{hour}"
  }

  # If you don't need separate values, remove them
  remove_field => ["year", "month", "day", "hour"]
}