[SOLVED] Add log rotation support to airflow logs folder

Add log rotation support to airflow logs folder

I want to automate deleting all the log files(both scheduler and dag logs) that are older than 7 days.

The logs folder is mounted volume from the container -

  volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins

I tried adding the below log rotation related environment variables to docker compose based on official documentation, but that is not working as expected.

    AIRLOW__CORE__LOG_ROTATE: 'true'
    AIRFLOW__CORE__MAX_LOG_AGE_IN_DAYS: '7'

Does airflow support log rotation? Do we need to write a custom script to handle this?

Solution

If you use official Airflow Docker Image it should contain clean-logs script, so you might add this service into your docker-compose file

  airflow-log-groomer:
    <<: *airflow-common
    environment:
      <<: *airflow-common-env
      CONNECTION_CHECK_MAX_COUNT: "0"
      AIRFLOW__LOG_RETENTION_DAYS: "7"
    command:
      - bash
      - /clean-logs