I am using the Splunk OTEL Collector Helm chart to send logs from my GKE pods to the Splunk Cloud Platform. I have set UsesplunkIncludeAnnotation
to true
to filter logs from specific pods. This setup was working fine until I tried to filter the logs being sent. I added the following configuration to my splunk
values.yaml:
config:
processors:
filter/ottl:
error_mode: ignore
logs:
log_record:
- 'IsMatch(body, "GET /status")'
- 'IsMatch(body, "GET /healthcheck")'
When I applied this configuration, the specified logs were excluded as expected, but it did not filter logs from the specified pods. I am still receiving logs from all my pods, and the annotation is not taking effect. Additionally, the host is not displaying correctly and is showing as "unknown".
My questions are:
The issue seems to be related to the placement of the processors within the service pipeline.
Here’s how you can address the concerns:
The filter/ottl processor needs to be correctly placed within the service pipeline to apply log exclusions effectively. Ensure that this processor is included in the pipeline configuration.
The issue with the host being displayed as "unknown" can be addressed by adding the resourcedetection processor to your pipeline. This processor helps in detecting and adding resource-related attributes to the logs.
config:
processors:
filter/ottl:
logs:
log_record:
- 'IsMatch(body, "GET /status")'
- 'IsMatch(body, "GET /healthcheck")'
service:
pipelines:
logs:
processors:
- filter/ottl
- resourcedetection
- k8sattributes
- batch
- filter/logs
- resource/logs
This setup should effectively filter the logs as intended and include the correct pods based on annotations.