logstashlogstash-forwarderfilebeat

Tags index with filebeat and logstash


I use logstash-forwarder and logstash and create a dynamic index with tags with this configuration:

/etc/logstash/conf.d/10-output.conf

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "logstash-%{tags}-%{+YYYY.MM.dd}"
  }
}

/etc/logstash-forwarder.conf

"files": [
    {
      "paths": [
        "/var/log/httpd/ssl_access_log",
        "/var/log/httpd/ssl_error_log"
       ],
      "fields": { "type": "apache", "tags": "mytag" }
    },

The associated filebeat configuration is:

/etc/filebeat/filebeat.yml

filebeat:
  prospectors:
    -
     paths:
       - /var/log/httpd/access_log
     input_type: log
     document_type: apache
     fields:
       tags: mytag

In Kibana, instead of mytag I see beats_input_codec_plain_applied on all of my indices.


Solution

  • I have resolved inserting a filter to logstash:

    filter {
        if "beats_input_codec_plain_applied" in [tags] {
            mutate {
                remove_tag => ["beats_input_codec_plain_applied"]
            }
        }
    }