airflowprometheusstatsd

airflow statsd prometheus mapping rules

I have tried to use statsd_exporter as a bridge to export airflow metrics to prometheus.

Airflow.cfg:

statsd_on = True
statsd_host = localhost
statsd_port = 9132
statsd_prefix = airflow

prometheus.yml

  - job_name: 'airflow'
    scrape_interval: 5s
    static_configs:
    - targets: ['localhost:9135']

start statsd_exporter as:

 ./statsd_exporter --web.listen-address=":9135" --statsd.listen-tcp=":9132" --log.level=debug

Seems mapping rule is missing as there is no airflow related metrics got in prometheusscreen

Would someone have any sample mapping rule for airflow statsd_exporter to prometheus?

Solution

  • You can find mappings in the official repo https://github.com/apache/airflow/blob/master/chart/dockerfiles/statsd-exporter/mappings.yml

    There's also this slightly tweaked version which drops unmatched metrics https://github.com/astronomer/ap-vendor/blob/main/statsd-exporter/include/mappings.yml