parsingfluentdfluent-bit

What's the purpose of `Time_Key` in Fluent Bit parsers?


I'm new to learning Fluent Bit, and I can't wrap my head around the benefit of specifying the Time_Key field in a parser.

An example from the documentation is below, but I don't know what the point of defining this is. By declaring Time_Key and a Time_Format it seems to take your timestamp field out of your log data and does what with it?

When I added this option to my own parser.conf and sent the logs to Splunk, my timestamp field doesn't even appear in Splunk, whereas it did before I added Time_Key. I know you can set the Time_Keep option to bring it back, but why would it remove a timestamp field from your log in the first place? Timestamp is something that seems useful to me. So, what am I missing?

From the documentation,


[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S %z

The following log entry is a valid content for the parser defined above:

{"key1": 12345, "key2": "abc", "time": "2006-07-28T13:22:04Z"}

After processing, its internal representation will be:

[1154103724, {"key1"=>12345, "key2"=>"abc"}]

The time has been converted to Unix timestamp (UTC) and the map reduced to each component of the original message.


I have added the Time_Key field to my parser and have seen it remove the timestamp field from my log data. I just don't know what the purpose of doing this is?


Solution

  • If you don't use `Time_Key' to point to the time field in your log entry, Fluent-Bit will use the parsing time for its entry instead of the event time from the log, so the Fluent-Bit time will be different from the time in your log entry.

    If you use Time_Key and Fluent-Bit detects the time, Fluent-Bit will drop the original field. You can keep it by setting Time_Keep On in your parser.conf, which will result in duplicate data.