How to install a new Jupyter Kernel from script

I am trying to install a new kernel on a jupyter environment from bash script.

In a dockerfile that uses jupyter/minimal-notebook as the parent image, I run a bash script that:

• generates a new virtual env: python -m venv new_kernel
• installs numpy and ipykernel new_kernel/bin/pip install ipykernel numpy
• registers the kernel: new_kernel/bin/python -m ipykernel install --name new_kernel --display-name "Python (new_kernel)".

Unfortunately, as a result I get:

• the new label "Python (new_kernel)" available among the kernels, BUT if I use that kernel it does not have numpy installed
• investigating in the new_kernel/lib/python3.9/site-packages folder I find numpy installed. This makes me think that the kernel is not being seen somehow.

How can I get the new kernel working along with the libraries installed?

I understood how it works, so I answer myself, to the benefit of those who will have the same problem.

TL;DR;

The environment is loaded fine. Brutally import your new kernel libraries and you should have no problems.

As far as I understand, the cell bash emulator acts on a different and independent context from the python context that is loaded into the notebook. So best to use sys.prefix and list(pkg_resources.working_set) to inspect the environment.

Therefore you can install a new kernel starting with a script like this

python -m venv new_kernel
new_kernel/bin/pip install ipykernel numpy
new_kernel/bin/python -m ipykernel install --name new_kernel --display-name "Python (new_kernel)"

Reproducing the case

python -m venv new_kernel
# a new new_kernel folder has been created, containing the environment

# install ipykernel and numpy in the new env
new_kernel/bin/pip install ipykernel numpy

# install the new kernel tied to the new env
new_kernel/bin/python -m ipykernel install --name new_kernel --display-name "Python (new_kernel)"

# launch jupyter lab
jupyter lab

From the jupyterlab GUI you should see new_kernel among the available kernels. Click it in order to open a new notebook with it already loaded.

The problem

Let's inspect the environment.

• If you type !which python in a new cell we get a different path to the new environment.

• If you type !pip list, we do not see numpy installed

but

• By typing the following instructions we find that the environment path has been correctly loaded

import sys
print(sys.prefix)
# should see something like /home/jovyan/new_kernel

print(sys.path)

• If you inspect the packages loaded in this notebook you'll find numpy installed!

import pkg_resources
list(pkg_resources.working_set)

What is happening?

We have just seen that if we run commands from the bash emulator (!<command>) of the cell the results are lies; whereas if we inspect the python context of the working kernel we find that the environment has been correctly loaded.

Unfortunately, I have found no official references to this phenomenon so far, and I invite anyone to contribute to this question. My personal explanation is that the bash emulator accesses the same CLI that launched jupyter lab, which can be in any path and with any other environment on/off; while the notebook launched by jupyter lab actually loads the context from the kernel configuration.

So they act on two different and independent contexts, so it is better to inspect the python context directly.

Side note

Interestingly, this problem is not reproduced if you use nb_conda_kernels, since it uses a custom KernelSpecManager.