AWS provides to run Airflow as a service. It has 3 different environments in which it can be run, namely Small, Medium and Large. These environments have different configurations of CPU and RAM. I used the Large config which has 4vCPUs and 8GB RAM (for each worker) which allows 20 concurrent tasks to be run by default (by each worker). I was able to run around 40 tasks in parallel with 1 worker(Min. worker = 1, Max Worker = 1, Scheduler =2)(each task was of a different dag run). When I run more than 40 tasks I find that the latency of the task increases significantly. So when I increase the number of workers the number of tasks I am able to run should be a multiplication of the tasks by one worker and number of workers, but with 25workers( I was only able to run around 260 tasks. When I go beyond this value the latency increases.
I am trying to figure out what other config changes can be tweaked to increase the performance and reduce latency. Please let me know if you need more details.
I've attached the screenshot of my config.
Apparently the max number of schedulers that AWS can maintain feasibly is 5. The number of tasks airflow could pick was more dependant on the schedulers in my use case since the tasks were not compute intensive. So the number of schedulers needed to schedule the tasks were more critical as the tasks were completed quickly and idle workers to take up next tasks were available. Since the max number of schedulers is 5, this is the max airflow can do for my use case and hence not suitable for my application.
