I have a json event coming into logstash that looks like (values are different in each event):
{
"metadata":{
"app_type":"Foo",
"app_namespace":"System.Bar"
"json_string":"{\"test\":true}"
}
}
I would like my logstash filter to output the following to elasticsearch (i'm not worried about trimming the existing fields):
{
"app_event":{
"foo":{
"system_bar":{
"test":true
}
}
}
}
I cannot figure out the syntax for how to use field values as the names of new fields.
I tried a few different ways, all resulted in a config error when starting my logstash instance. I'm only including the filter block of logstash.conf
in my examples, the rest of the config is working as expected.
1.
# note this one does not have replacement and lowercase values I want
filter {
json {
source => "[metadata][json_string]"
target => "[app_event][%{[metadata][app_type]}][%{[metadata][app_namespace]}]"
}
}
filter {
appType = "%{[metadata][instrument_type]}".downcase
appNamespace = "%{[metadata][app_namespace]}".downcase
appNamespace.gsub(".", "_")
json {
source => "[metadata][json_string]"
target => "[app_event][#{appType}][#{appNamespace}]"
}
}
filter {
json {
source => "[metadata][json_string]"
target => "[app_event]"
}
}
The json filter does not sprintf the value of target, so you cannot use a json filter in those ways. To make the field name variable you must use a ruby filter. Try
json { source => "message" target => "[@metadata][json1]" remove_field => [ "message" ] }
json { source => "[@metadata][json1][metadata][json_string]" target => "[@metadata][json2]" }
ruby {
code => '
namespace = event.get("[@metadata][json1][metadata][app_namespace]")
if namespace
namespace = namespace.downcase.sub("\.", "_")
end
type = event.get("[@metadata][json1][metadata][app_type]")
if type
type = type.downcase
end
value = event.get("[@metadata][json2]")
if type and namespace and value
event.set("app_event", { "#{type}" => { "#{namespace}" => value } } )
end
'
}