[SOLVED] ModuleNotFoundError on worker when attempting to import local module #222

ModuleNotFoundError on worker when attempting to import local module #222

I am getting a ModuleNotFound error on the worker Pods when attempting to import a local module. I'm able to import without issue on the master node. The master and workers are using the exact same image.

The working dir on the image is organized as follows:

WorkDIR/
       TheNotebook.ipynb
       MyModule/
                __init__.py
                myModule.py

The notebook is similar to

from MyModule import workerFunc

fut = daskClient.submit(workerFunc,*workerArgs);
fut.result()

which yields:

/usr/local/lib/python3.6/site-packages/distributed/client.py in result(self, timeout)
    222         if self.status == "error":
    223             typ, exc, tb = result
--> 224             raise exc.with_traceback(tb)
    225         elif self.status == "cancelled":
    226             raise result

/usr/local/lib/python3.6/site-packages/distributed/protocol/pickle.py in loads()
     57 def loads(x):
     58     try:
---> 59         return pickle.loads(x)
     60     except Exception:
     61         logger.info("Failed to deserialize %s", x[:10000], exc_info=True)

ModuleNotFoundError: No module named MyModule

I don't "install" the module in the docker image, I'm just assuming the working directory on the worker pod is the same as the WORKDIR on the docker image. Is this incorrect? Do I need to create a setup.py and install MyModule?

Thanks!

Solution

It sounds like you do not have the current directory set in your PYTHONPATH on your workers.

You'll need to do something like this in your Dockerfile:

ENV PYTHONPATH ".:${PYTHONPATH}"

This will check the current directory for modules before checking others in your path. Therefore it can also be used to override installed libraries.

See How do you add a path to PYTHONPATH in a Dockerfile for more information.