I want to use Apache Airflow for IoT device simulator written in python. Let's say I want to run that script for 25000 devices and the 25000 invocations should run concurrently. Is apache airflow is right tool to be used or there are other opensource tools which will be good fit in this use case?
Apache Airflow is not suitable for running 25,000 concurrent IoT device simulations because it is designed for orchestrating batch workflows with scheduled, heavyweight tasks not for high concurrency, real time, or event driven execution. Open source tools like Ray or Temporal are a better fit to execute the simulations in parallel, while using InfluxDB to collect and store the time-series data generated by each simulated device. This setup allows efficient simulation execution and real time data analysis.