prefect

Dependent Prefect flow runs across multiple instances


I have two physical machines. I would like Prefect to automatically execute a flow run on machine B once a flow run on machine A finishes. What's the most lightweight way to do this?

In other words, I want to run a flow of flows, in which the flows are executed on different physical devices.

Why do I need this? Some of the data is sensitive so they are only accessible on a particular machine.


Solution

  • The most lightweight version would be to leverage the orchestrator pattern - the same applies to both Prefect 1 and Prefect 2.

    For Prefect 2, check out: https://discourse.prefect.io/t/how-to-create-a-flow-run-from-deployment-orchestrator-pattern/803

    For Prefect 1, check out: https://discourse.prefect.io/t/can-i-run-a-flow-of-flows-that-triggers-each-child-flow-on-a-different-machine/180