loggingairflowdatadog

Datadog exclude the logs

I am trying to exclude Info-related logs on Datadog from airflow but this is not helping. Any thoughts if I am doing something wrong?

ad.datadoghq.com/worker.logs: '[{"source": "airflow","service":"airflow",
      "log_processing_rules": [{"type": "exclude_at_match", "name":
      "exclude_health", "pattern" : "\bINFO\b" }]}]'

Below is the sample log from Datadog

[2024-08-09T20:04:03.531+0000] {main.py:244} INFO - data sent from 2024-07-23 00:00:00 to 2024-07-30 00:00:00

Similarly, I tried for a health check with pattern "\bhealth\b" for the below log and that didn't work either

345.0.0.5 - - [03/Aug/2023:20:10:33 +0000] "GET /airflow/api/v1/health HTTP/1.1" 200 375 "-" "Datadog Agent/7.54.0" ```
Solution

  • Datadog uses Golang regex syntax for matching patterns in logs.

    \b in Golang represents a backspace and not a word boundary.

    You need to escape twice to make the word boundary and create the matching pattern

     "pattern" : "\\bINFO\\b"