[SOLVED] How to run a jupyter notebook using a singularity image?

How to run a jupyter notebook using a singularity image?

I have built a singularity image (jupyter.sif) that contains the latest anaconda version and I want to use that to run a jupyter notebook.

I would like to run something like this:

singularity exec \
  --bind /path/outside/image/:/path/inside/image/ \
  jupyter.sif jupyter notebook \
    --notebook-dir=/path/to/dir --no-browser --ip=127.0.0.1

Essentially, just launch a normal jupyter notebook that will use the python version and packages installed in the singularity image. Ideally the notebook will read and write files outside the image (hence why I specify a bind path).

However, when I run the above command I get the following error:

> Traceback (most recent call last):   File
> "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line
> 528, in get
>     value = obj._trait_values[self.name] KeyError: 'runtime_dir'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
> 
>   File "/opt/conda/bin/jupyter-notebook", line 11, in <module>
>     sys.exit(main())
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py",
> line 266, in launch_instance
>     return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py",
> line 657, in launch_instance
>     app.initialize(argv)
> 
>   File "<decorator-gen-7>", line 2, in initialize
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py",
> line 87, in catch_config_error
>     return method(app, *args, **kwargs)
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line
> 1626, in initialize
>     self.init_configurables()
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line
> 1319, in init_configurables
>     connection_dir=self.runtime_dir,
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line
> 556, in __get__
>     return self.get(obj, cls)
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line
> 535, in get
>     value = self._validate(obj, dynamic_default())
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py",
> line 99, in _runtime_dir_default
>     ensure_dir_exists(rd, mode=0o700)
> 
>   File
> "/opt/conda/lib/python3.7/site-packages/jupyter_core/utils/__init__.py",
> line 13, in ensure_dir_exists
>     os.makedirs(path, mode=mode)
> 
>   File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
>     makedirs(head, exist_ok=exist_ok)
> 
>   File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
>     makedirs(head, exist_ok=exist_ok)
> 
>   File "/opt/conda/lib/python3.7/os.py", line 221, in makedirs
>     mkdir(name, mode)
> 
> OSError: [Errno 30] Read-only file system: '/run/user'

I assume this is because the notebook is trying to write stuff inside the image and since i don't run the image as sudo (i don't want to do this) and i also do not specify --writable when executing the image (also don't want to do this), it complains about not being able to write within the image.

Is there a way of "telling" the image to read and write files in my home directory and not inside the image?

Thanks

Solution

I just came across the same problem and this seems to have solved it for me: map /run/user inside of the container to your working directory outside (or a directory of your choosing). For your case, would mean probably this:

singularity exec --bind /path/outside/image/:/path/inside/image/ --bind $PWD:/run/user jupyter.sif jupyter notebook --notebook-dir=/path/to/dir --no-browser --ip=127.0.0.1

When running like this, in my case a folder 1000 was created