daskdask-distributed

What is the cleanest way to detect whether I'm running in a dask Worker


Given a dask.distributed cluster, for example a LocalCluster, what is the most robust way to detect if I'm running a python code from within a Worker instance?

This can be code that is not strictly associated with a task, for example while setting up the Worker's environment

The answer to this previous question Using Dask distributed, how can a function introspect to know if it's running as a distributed future or as a "normal" function? does not work anymore given the recent changes to using contextvars


Solution

  • You probably want distributed.worker.get_worker() - this will raise an exception if the code is not running within a worker. Previously, this would only raise if no worker was defined, which is often, but not necessarily the same thing.

    If you want to know the latter thing (which the question suggests), you can look at distributed.worker.Worker._instances, and if any of the weak references there are still alive.