pythonenvironment-variablescommand-line-interfacenetflix-metaflow

How to set MetaFlow's --max-workers flag from within Python definition?

MetaFlow permits you to set the maximum number of concurrent tasks using the --max-workers CLI flag (ref: https://docs.metaflow.org/metaflow/scaling#safeguard-flags). However, I would like to avoid setting this every time.

Is it possible to set the --max-workers flag from the Python definition of the FlowSpec (without the CLI)?

Solution

  • The CLI flag sets the METAFLOW_RUN_MAX_WORKERS environment variable. You can set this within Python by setting this environment variable before you define your FlowSpec:

    # force MetaFlow to use only 1 concurrent task during a foreach fanout
os.environ.update({"METAFLOW_RUN_MAX_WORKERS": "1"})

    Source: chat with @Savin