fluentd

Difference between timekey and flush_interval

I configured a fluentd forwarder service to collect logs of multiple fluentd instance and forward collected to an Elasticsearch cluster.

I configured the forwarder to buffer logs in chunks:

<match **>
  @type elasticsearch
  ...
  <buffer time>
    @type file
    ...
    flush_interval 30s
    timekey 30
    ...
  </buffer>

On this part I got confused about the usage of timekey and flush_interval. The documentation sounds very similar between the two. For timekey:

Output plugin will flush chunks per specified time (enabled when time is specified in chunk keys)

and for flush_interval:

flushes/writes chunks per specified time via flush_interval

Do I only need one of those specified or both?

Solution

  • From https://docs.fluentd.org/configuration/buffer-section

    If the argument time and the parameter timekey (required) are specified, the output plugin writes events into chunks grouped by time key.

    And for flushing:

    Following are the flushing parameters for chunks to optimize performance (latency and throughput)

    So in my understanding:

    The timekey serves for grouping data in chunks by time, but not for saving/sending chunks.

    The flush_interval defines how often the prepared chunk will be saved to disk/memory.