kubernetesfilterkubernetes-helmsplunksplunk-cloud

Exclude unnecessary logs being sent to splunk cloud platform


I am using the Splunk OTEL Collector Helm chart to send logs from my GKE pods to the Splunk Cloud Platform. I have set UsesplunkIncludeAnnotation to true to filter logs from specific pods. This setup was working fine until I tried to filter the logs being sent. I added the following configuration to my splunk values.yaml:

  config:
    processors:
      filter/ottl:
        error_mode: ignore
        logs:
          log_record:
            - 'IsMatch(body, "GET /status")'
            - 'IsMatch(body, "GET /healthcheck")'

When I applied this configuration, the specified logs were excluded as expected, but it did not filter logs from the specified pods. I am still receiving logs from all my pods, and the annotation is not taking effect. Additionally, the host is not displaying correctly and is showing as "unknown".

My questions are:

  1. How can I exclude these specific logs more effectively?
  2. Is there a more efficient way to achieve this filtering?
  3. Can I do it in Splunk Cloud UI?

Solution

  • The issue seems to be related to the placement of the processors within the service pipeline.

    Here’s how you can address the concerns:

    The filter/ottl processor needs to be correctly placed within the service pipeline to apply log exclusions effectively. Ensure that this processor is included in the pipeline configuration.

    The issue with the host being displayed as "unknown" can be addressed by adding the resourcedetection processor to your pipeline. This processor helps in detecting and adding resource-related attributes to the logs.

    config:
      processors:
        filter/ottl:
          logs:
            log_record:
              - 'IsMatch(body, "GET /status")'
              - 'IsMatch(body, "GET /healthcheck")'
      service:
        pipelines:
          logs:
            processors:
              - filter/ottl
              - resourcedetection
              - k8sattributes
              - batch
              - filter/logs
              - resource/logs
    

    This setup should effectively filter the logs as intended and include the correct pods based on annotations.