fluent-bit

Is it possible to use a fluent-bit record's timestamp?

I'm trying to create a fluent-bit config which uses a record's timestamp to a custom key using a filter. Something like:

[INPUT]
  Name tail
  Path /some/path
  ... 

[FILTER]
  Name record_modifier
  Match *
  Record fluenbit_orig_ts SOME_MAGIC_WAY_TO_GET_UNIXTIME

[OUTPUT]
  Name           stdout
  Match          *

The rationale for this that I'm using several parsers, each has its own time format (Time_Format, as it's used in the regular expression parser. Even if I used Time_Keep, it won't help because the time is specified differently by different services, with a different Time_Format). I'd like the records getting to an [OUTPUT} to have the same key to describe the timestamp. In this example that key would be fluenbit_orig_ts

Is this possible?

Solution

  • I got an answer from the fluent-bit slack channel.

    Seems like this is possible using lua filters. Specifically, this example seems to be relevant: https://github.com/fluent/fluent-bit/blob/master/scripts/append_tag.lua