logstashelk

Why my field reference with datastream output in logstash is not working?


I'm trying to create logstash.conf to send mylog app to elk stack. I'm using field reference in ouput tag config to create a dynamic stream data, and it return me an error then crash the logstash:

Badly formatted index, after interpolation still contains placeholder: [logs-%{[@metadata][ds_dataset]}-default];

Im dont know is logstash suport field reference in datastream or not, but using with index data conf its ok.

logstash.conf

input {
  beats {
    port => 5044
  }
}

filter {
  # Gán các field cần dùng vào @metadata
  mutate {
    add_field => {
      "[@metadata][ds_dataset]" => "test"
      # Nếu bạn muốn dynamic namespace thì bật dòng dưới
      # "[@metadata][ds_namespace]" => "%{[fields][namespace]}"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }

  elasticsearch {
    hosts => ["http://elasticsearch:9200"]

    data_stream => true
    data_stream_type => "logs"
    data_stream_dataset => "log-%{[@metadata][ds_dataset]}"
    data_stream_namespace => "default"  # Có thể đổi thành: %{[@metadata][ds_namespace]}
  }
}


Solution

  • The elasticsearch output does not support arbitrary field references in the datastream name. That is, the code does not sprintf the parts of the name.

    However, it does support using a field (hash) called [data_stream] that contains [type], [dataset] and [namespace] items. This is enabled using the data_stream_auto_routing option on the output.

    So your filter would include

    mutate {  
         add_field => {  
             "[data_stream][dataset]" => "test"  
             "[data_stream][namespace]" => "%{[fields][namespace]}"  
             "[data_stream][type]" => "logs"  
         }  
    }  
    

    Note that you must make sure [fields][namespace] exists before calling this filter, or you will get an error about the index name.