pythonjupyter-notebookjupyterpythonpath

Minimal way to set PYTHONPATH for developing python in VS Code and Jupyter notebook server


Related questions that do not apply:


I'm helping to organize some python code with a team at work. For "reasons" the project has multiple subdirectories and so needs PYTHONPATH or equivalent to add those subdirectories to work. For example, with this project setup…

project/
   .venv/
   foo/
      bar.py
   jim/
      jam.py

…we want Jupyter notebooks to work for the code:

import jim
import jam

We want these notebooks to work both (a) within VS Code, and (b) when starting a Jupyter notebook server and connecting to it from Google Colab.

Desires:

  1. Little-to-no setup work needed per user. (Ideally, they just clone the project and run a launch command or manual script from within the .venv to get Jupyter server running.)
  2. No absolute paths hard-coded anywhere. (Multiple copies of the project sometimes exist on disk, are added and removed; a user needs to be able to launch Jupyter notebook servers for various locations.)
  3. Specify the foo and bar paths in as few places as possible; ideally one. (There are many, many places where extra paths can be specified within the ecosystem of Python, Jupyter, VS Code, pytest, pylint, etc.)
  4. No need to pollute every single notebook file by adding sys.path modifications at the top of each.
  5. Live development is possible. The developers are not installing this project and then using the notebooks, they are iteratively using the notebooks and modifying core library code that affects the next notebook evaluation.

Within VS Code I've gotten the notebooks working by:

  1. Adding .env file at the project root with the contents:

    PYTHONPATH=foo:bar
    
    • The Python extension has a default setting:
      "python.envFile" : "${workspaceFolder}/.env"
  2. Ensuring all notebooks are run (in VS Code) from the root directory by changing the setting jupyter.notebookFileRoot from the default ${fileDirname} to ${workspaceFolder}.


I've also had to add the locations in the pyproject.toml file for testing:

[tool.pytest.ini_options]
pythonpath = [
    "foo",
    "bar",
]

Now…how can I make it so that all the developers can launch a Jupyter notebook server for the project, inside the project's venv, that gets PYTHONPATH set correctly? It can be a single launch_jupyter.sh script, but I'd prefer not to have to maintain the same list of foo:bar in that script file. (Because it's not DRY, and the actual directory names are longer and more than just a couple.)


Solution

  • I ended up writing a shell script that:

    1. Activates (or creates) the virtual environment, using uv.
    2. Loads environment variables from the same .env file that VSCode uses
    3. Converts all the relative paths in PYTHONPATH to absolute
    4. Starts the Jupyter server.

    Now developers can just ./start-jupyter-server.sh and then connect to it from Google Colab.

    #!/bin/bash
    
    # Find the directory of this file
    SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" &> /dev/null && pwd)
    
    # Path to the desired virtual environment
    DESIRED_VENV="$SCRIPT_DIR/.venv"
    
    # Check if a virtual environment is active
    if [ -n "$VIRTUAL_ENV" ]; then
      # If active, check if it's the desired one
      if [ "$VIRTUAL_ENV" != "$DESIRED_VENV" ]; then
        echo "Deactivating virtual environment $VIRTUAL_ENV because it is not $DESIRED_VENV..."
        deactivate
      fi
    fi
    
    # Activate the desired virtual environment
    if [ ! -d "$DESIRED_VENV" ]; then
      echo "Virtual environment not found at $DESIRED_VENV; creating."
      uv venv
    fi
    
    echo "Activating virtual environment..."
    source "$DESIRED_VENV/bin/activate"
    uv sync
    
    # Load environment variables from .env file
    if [ -f "$SCRIPT_DIR/.env" ]; then
      # Enable exporting variables from .env file
      set -a
      # Source the .env file
      source "$SCRIPT_DIR/.env"
      # Disable exporting variables
      set +a
    else
      echo "Error: .env file not found."
      exit 1
    fi
    
    # Convert relative paths to absolute paths
    PYTHONPATH=$(echo "$PYTHONPATH" | tr ':' '\n' | while read -r path; do
      echo -n "$SCRIPT_DIR/$path:"
    done | sed 's/:$//')
    
    # Set default port if not provided
    PORT=${1:-8888}
    
    # Run Jupyter notebook
    jupyter notebook \
      --NotebookApp.allow_origin='https://colab.research.google.com' \
      --Application.log_level=50 \
      --port=$PORT \
      --NotebookApp.port_retries=0 \
      --no-browser