I'm testing Fluent-bit for my local k8s cluster which has a CRI runtime interface and I'm sending logs to a slack channel. But the problem is that Fluent-Bit is assigning a "timestamp" in the log and I'm not able to remove it. Maybe someone knows a solution?
Here is the ConfigMap of my Fluent-Bit:
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging1
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 2
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input-kubernetes.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE output-syslog.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*
Parser cri
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
filter-kubernetes.conf: |
output-syslog.conf: |
[OUTPUT]
Name slack
Match *
webhook [LINK]
parsers.conf: |
[PARSER]
Name cc
Format regex
Format cri
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Also here is the raw log coming from my app:
2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -
And this is the log that is sent to Slack:
["timestamp": 1681315742.016981904, {"log"=>"2023-04-12T16:09:02.016483996Z stderr F 10.244.0.1 - - [12/Apr/2023 16:09:02] "GET / HTTP/1.1" 200 -"}]
I've used different Filters and Parsers. Currently my Fluent-bit is using the latest image
The parser cri does not exists in your configuration, therefore the files are not parsed correctly and you receive "2023-04-12T16:09:02.016483996Z stderr F " as part of your message log. Just use the official parsers.conf provided by fluent-bit or fix your typos (Name: cri, not cc, Format is regex).
A valid snipped would be:
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z