We are observing Airflow is sending large amount of logs to Datadog and we want to restrict/Reduce these logs by excluding logs from below tasks:
Do we have any configuration settings where I can define this requirement?
We have Airflow-2.0 running on Kubernetes.
To send only a specific subset of logs to Datadog, use the log_processing_rules parameter in your configuration file with the exclude_at_match or include_at_match type. Apply this to your schedular which generating the logs. Refer to this Datadog official doc for more information.