I'm trying to create logstash.conf to send mylog app to elk stack. I'm using field reference in ouput tag config to create a dynamic stream data, and it return me an error then crash the logstash:
Badly formatted index, after interpolation still contains placeholder: [logs-%{[@metadata][ds_dataset]}-default];
Im dont know is logstash suport field reference in datastream or not, but using with index data conf its ok.
logstash.conf
input {
beats {
port => 5044
}
}
filter {
# Gán các field cần dùng vào @metadata
mutate {
add_field => {
"[@metadata][ds_dataset]" => "test"
# Nếu bạn muốn dynamic namespace thì bật dòng dưới
# "[@metadata][ds_namespace]" => "%{[fields][namespace]}"
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["http://elasticsearch:9200"]
data_stream => true
data_stream_type => "logs"
data_stream_dataset => "log-%{[@metadata][ds_dataset]}"
data_stream_namespace => "default" # Có thể đổi thành: %{[@metadata][ds_namespace]}
}
}
The elasticsearch output does not support arbitrary field references in the datastream name. That is, the code does not sprintf the parts of the name.
However, it does support using a field (hash) called [data_stream] that contains [type], [dataset] and [namespace] items. This is enabled using the data_stream_auto_routing option on the output.
So your filter would include
mutate {
add_field => {
"[data_stream][dataset]" => "test"
"[data_stream][namespace]" => "%{[fields][namespace]}"
"[data_stream][type]" => "logs"
}
}
Note that you must make sure [fields][namespace] exists before calling this filter, or you will get an error about the index name.