I want to create a system that automates virtual machines to do various different tasks. Every few minutes I want to trigger a workflow that powers up a vm, run a process inside, and shut it down. I’m considering two frameworks for such a system, temporal and Apache airflow. The workflows change frequently and each task might take from a few minutes to several hours.
Which one should I choose? Or maybe there are other better options?
One major issue is to pass the pyvmomi connection object between tasks / activities. How should I tackle this, since I need the ability to manipulate the vm at every stage of the workflow?
My recommendation would be to simply choose one, create an architectural spike and see if it fits.
Both products more-or-less fit the same category. If you have a team knowledge of one over the other I'd go for that, otherwise the spike should suffice.
Remember, one of the biggest hurdles in software development is getting to the "perfect" product or choice of language. The same can be said on choosing a backend language such as golang, Python or NodeJS - the differences at the start are relatively minor. Eventually, time spent deciding and over-engineering is time that you don't use for active development. By spiking, you will get considerably more knowledge, similar to releasing an alpha version and requesting user feedback.
Bottom line: If one system suddenly doesn't suite your needs, the spike will allow you to easily know why, and switch accordingly before the full development.