fluent-bit

How to remove "timestamp" date from the Fluent-Bit logs?


I'm testing Fluent-bit for my local k8s cluster which has a CRI runtime interface and I'm sending logs to a slack channel. But the problem is that Fluent-Bit is assigning a "timestamp" in the log and I'm not able to remove it. Maybe someone knows a solution?

Here is the ConfigMap of my Fluent-Bit:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging1
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         2
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
 
    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-syslog.conf
 
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On

  filter-kubernetes.conf: |

  output-syslog.conf: |
    [OUTPUT]
        Name               slack
        Match              *
        webhook            [LINK]
        
 
  parsers.conf: |
    [PARSER]
        Name          cc
        Format        regex
        Format        cri
        Regex         ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key      time
        Time_Format   %Y-%m-%dT%H:%M:%S.%L%z

Also here is the raw log coming from my app:

2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -

And this is the log that is sent to Slack:

["timestamp": 1681315742.016981904, {"log"=>"2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -"}]

I've used different Filters and Parsers. Currently my Fluent-bit is using the latest image


Solution

  • The parser cri does not exists in your configuration, therefore the files are not parsed correctly and you receive "2023-04-12T16:09:02.016483996Z stderr F " as part of your message log. Just use the official parsers.conf provided by fluent-bit or fix your typos (Name: cri, not cc, Format is regex).

    A valid snipped would be:

    [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z