How to attach Job Labels to Job Events in Fluent Bit Kubernetes Events

I'm using Fluent Bit's kubernetes_events plugin to collect Kubernetes events directly from the API server. I'm aiming to ingest these events into an OpenSearch cluster.

A key requirement is to enrich each event with the labels of the Kubernetes object that triggered it. Specifically, for Job events, I need to include the labels defined in the Job's configuration.

I attempted to achieve this by configuring the kubernetes_events plugin and the kubernetes Filter with the expectation that it would automatically attach the labels. However, this didn't work as anticipated.

Here's the config map that I used. I'm using rancher desktop for the kubernetes test enviroment

[SERVICE]
    flush           1
    log_level       info

[INPUT]
    name            kubernetes_events
    tag             k8s_events
    kube_url        https://kubernetes.default.svc

[FILTER]
    Name rewrite_tag
    Match k8s_events
    Rule $involvedObject['kind'] ^(Job)$ $TAG.pod.$involvedObject['name'].$involvedObject['namespace'].event false

[FILTER]
    Name kubernetes
    Buffer_Size 32MB
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
    Keep_Log Off
    Match k8s_events.pod.*
    Owner_References On
    Kube_Tag_Prefix k8s_events.Job.
    Merge_Log On
    tls.verify Off
    Use_Kubelet true

# Output for job events
[OUTPUT]
    name            stdout
    match           k8s_events.Job.*
    format          json_lines

This displays the events as JSON but no labels about the Jobs.

What am I missing here?

Solution

If the labels are not showing up then it means maybe fluent bit’s kubernetes filter is not configured correctly. For this you need to manually enrich the events using a custom Lua filter if the default kubernetes metadata collection isn’t sufficient.

Regarding your query, whether it requires direct calls to the Kubernetes API server via Lua scripts. Yes, Lua plugin would require direct API calls to the k8s API server to fetch Job labels. But the Fluent Bit Lua filter plugin has some limitations, like the Lua plugin does not include the necessary HTTP modules to fetch job metadata from the k8s API. To resolve this you need to enrich the data through an external processor. Refer to this How to configure Fluent Bit to collect logs for your K8s cluster blog by Giulia Di Pietro, which will be helpful to resolve the issue.

Note: If you intend to use Lua to interact with kubernetes API directly you will need to implement HTTP requests within Lua however this may require additional modules that aren't included by default fluent bit’s Lua plugin.