I have a setup with Fluentd and Elasticsearch running on a Docker engine. I have swarms of services which I would like to log to Fluentd.
What I want to do is create a tag for each service that I run and use that tag as an index in Elasticsearch. Here's the setup that I have:
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match docker.service1>
@type elasticsearch
host "172.20.0.3"
port 9200
index_name service1
type_name fluentd
flush_interval 10s
</match>
<match docker.service2>
@type elasticsearch
host "172.20.0.3"
port 9200
index_name service2
type_name fluentd
flush_interval 10s
</match>
and so forth.
It would be annoying to have to include a new match tag for every single service I create, because I want to be able to add new service without updating my fluentd configuration. Is there a way to do something like this:
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match docker.**>
@type elasticsearch
host "172.20.0.3"
port 9200
index_name $(TAG)
type_name fluentd
flush_interval 10s
</match>
Where I use a $(TAG) variable to indicate that I want the Tag name to be the name of the index?
I've tried this from an answer I found here: ${tag_parts[0]}. This was printed literally as my index. So my index was "${tag_parts[0]}".
Thanks in advance.
I figured out that I needed to import the other Elasticsearch plugin. Here's an example of a match tag that I used:
<match>
@type elasticsearch_dynamic
host "172.20.0.3"
port 9200
type_name fluentd
index_name ${tag_parts[2]}
flush_interval 10s
include_tag_key true
reconnect_on_error true
</match>
I've imported the @elasticsearch_dynamic plugin instead of the @elasticsearch plugin. Then, I can use the ${tag_parts} thing.
The include_tag_key will include the tag in the json data.
It helps to read the documentation