airflowairflow-scheduler

Airflow: Concurrency Depth first, rather than breadth first?


In airflow, the default configuration seems to be to queue up tasks, in parallel, across days--from one day to the next.

However, if I spin this process up across, say, two years, then the airflow dag will churn through preliminary processes first, across all days, rather than taking, say, 4 days forward from start to finish concurrently.

How do I toggle airflow to execute tasks according to a depth first paradigm rather than a breadth first paradigm?


Solution

  • I have come across a similar situation. I used the following trick to achieve that depth-first behaviour.


    Explaination