deploymentdagster

Deploy to different machine profiles using DAGster


Can I specify the machine or machine type I'd like to deploy each task/workflow in Dagster separately?

(e.g. Having an A->B DAG, make step A performed on a machine of type M_a, and have step B occur on a machine of type M_b)

For example, if I have some CPU-heavy task and another which is GPU-heavy, I'll probably prefer they would not be run on the same machine type.


Solution

  • As of version 1.4.11, and for Amazon ECS, I didn't find a solution running in the scope of a task or a workflow, but in the scope of a deployment.

    That could be sufficient assuming flows for different machine types would be deployed separately. (And flows using several machine types may be broken into sub-flows)