python-3.xpip

pip install itself fails with "No module named 'distutils'"

I use Windows 11 and Python 3.12.5 64-bit, installed from https://www.python.org/downloads/

I'm trying to install requirements from some repo, but this fails:

PS C:\Data\Repos\myrepo> pip install -r .\requirements.txt
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\jonathanb\AppData\Local\Programs\Python\Python312\Scripts\pip.exe\__main__.py", line 4, in <module>
  File "C:\Users\jonathanb\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_internal\cli\main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\jonathanb\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_internal\cli\autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\jonathanb\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_internal\cli\main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "C:\Users\jonathanb\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_internal\cli\cmdoptions.py", line 18, in <module>
    from distutils.util import strtobool
ModuleNotFoundError: No module named 'distutils'

From several questions on this site (best one) I understand distutils was removed in Python 3.12, and if I want to use them I should run pip install setuptools. I tried that, and it failed the same way.

Question: Why does pip fail? How can I get it to work and restore requirements?

[Update - additional research] Newer pythons (tested with 3.13.1) come with a version of pip that doesn't depends on distutils (24.3.1). However, something on my machine (TBD: what?) installs an older version of pip (20.2.3). This version still uses distutils, and is therefore borked.

Workaround 1: uninstall+install Python, and then pip works - until something breaks it again Workaround 2: In C:\Users\%username%\AppData\Local\Programs\Python\Python313\Lib\site-packages, replace pip + pip-20.2.3.dist-info with pip + pip-24.3.1.dist-info from a working python installation.

Additional investigation: Running python with verbose, looks like I get different results regarding __pycache__:

Borked case:

PS C:\Users\jonathanb\Downloads\pip> python -v C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Scripts\pip.exe
(...)
# C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\__pycache__\__init__.cpython-313.pyc matches C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\__init__.py
# code object from 'C:\\Users\\jonathanb\\AppData\\Local\\Programs\\Python\\Python313\\Lib\\site-packages\\pip\\__pycache__\\__init__.cpython-313.pyc'
# C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\__pycache__\__init__.cpython-313.pyc matches C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\__init__.py
# code object from 'C:\\Users\\jonathanb\\AppData\\Local\\Programs\\Python\\Python313\\Lib\\site-packages\\pip\\_internal\\__pycache__\\__init__.cpython-313.pyc'
# C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\utils\__pycache__\__init__.cpython-313.pyc matches C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\utils\__init__.py

Working case:

PS C:\Users\jonathanb\Downloads\pip> python -v C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Scripts\pip.exe
(...)
# C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\__pycache__\__init__.cpython-313.pyc matches C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\__init__.py
# code object from 'C:\\Users\\jonathanb\\AppData\\Local\\Programs\\Python\\Python313\\Lib\\site-packages\\pip\\__pycache__\\__init__.cpython-313.pyc'
# C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\__pycache__\typing.cpython-313.pyc matches C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\typing.py
# code object from 'C:\\Users\\jonathanb\\AppData\\Local\\Programs\\Python\\Python313\\Lib\\__pycache__\\typing.cpython-313.pyc'
import '_typing' # <class '_frozen_importlib.BuiltinImporter'>
import 'typing' # <_frozen_importlib_external.SourceFileLoader object at _address_>
import 'pip' # <_frozen_importlib_external.SourceFileLoader object at _address_>
# C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\__pycache__\__init__.cpython-313.pyc matches C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\__init__.py# code object from 'C:\\Users\\jonathanb\\AppData\\Local\\Programs\\Python\\Python313\\Lib\\site-packages\\pip\\_internal\\__pycache__\\__init__.cpython-313.pyc'
                                                                                                                                                                 
# C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\utils\__pycache__\__init__.cpython-313.pyc matches C:\Users\jonathanb\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip\_internal\utils\__init__.py
# code object from 'C:\\Users\\jonathanb\\AppData\\Local\\Programs\\Python\\Python313\\Lib\\site-packages\\pip\\_internal\\utils\\__pycache__\\__init__.cpython-313.pyc'
import 'pip._internal.utils' # <_frozen_importlib_external.SourceFileLoader object at _address_>
Solution

  • Cause: A pre-commit hook was running python -m pip install --upgrade pip==20.2.3 - I'll get it fixed.

    How I found out

    After a few days the problem happened, and procmon showed a process running python -m pip install --upgrade pip==20.2.3. Looking in the (large) process tree, I saw that its parent process was running python (...)\scm\git\hooks/pre-commit