logstashlogstash-configurationlogstash-filter

Logstash Filter - How to use the value of a field as the name of a new field with parsed json?


I have a json event coming into logstash that looks like (values are different in each event):

{ 
  "metadata":{
     "app_type":"Foo",
     "app_namespace":"System.Bar"
     "json_string":"{\"test\":true}"
  }
}

I would like my logstash filter to output the following to elasticsearch (i'm not worried about trimming the existing fields):

{
  "app_event":{
    "foo":{
      "system_bar":{
        "test":true
      }
    }
  }
}

I cannot figure out the syntax for how to use field values as the names of new fields.

I tried a few different ways, all resulted in a config error when starting my logstash instance. I'm only including the filter block of logstash.conf in my examples, the rest of the config is working as expected.

1.

# note this one does not have replacement and lowercase values I want
filter {
   json {
      source => "[metadata][json_string]"
      target => "[app_event][%{[metadata][app_type]}][%{[metadata][app_namespace]}]"
   }
}
  1. I tried to create variables with values I need and then use those in the target
filter {
      appType = "%{[metadata][instrument_type]}".downcase
      appNamespace = "%{[metadata][app_namespace]}".downcase
      appNamespace.gsub(".", "_")

      json {
        source => "[metadata][json_string]"
        target => "[app_event][#{appType}][#{appNamespace}]"
      }      
}
  1. Just to confirm that everything else was set up correctly this does work, but does not include the dynamically generated field structure I'm looking for.
filter {
      json {
        source => "[metadata][json_string]"
        target => "[app_event]"
      }      
}

Solution

  • The json filter does not sprintf the value of target, so you cannot use a json filter in those ways. To make the field name variable you must use a ruby filter. Try

        json { source => "message" target => "[@metadata][json1]" remove_field => [ "message" ] }
        json { source => "[@metadata][json1][metadata][json_string]" target => "[@metadata][json2]" }
        ruby {
            code => '
                namespace = event.get("[@metadata][json1][metadata][app_namespace]")
                if namespace
                    namespace = namespace.downcase.sub("\.", "_")
                end
                type = event.get("[@metadata][json1][metadata][app_type]")
                if type
                    type = type.downcase
                end
                value = event.get("[@metadata][json2]")
                if type and namespace and value
                    event.set("app_event", { "#{type}" => { "#{namespace}" => value } } )
                end
            '
        }