I run the following command to install a package from a private Gitlab repo:
pip install -r requirements.txt --index-url https://username:password@private.gitlab.internal --cert /usr/local/share/ca-certificates/cert.crt
This works, but now I want to do it in a Dockerfile so that it can work in my Gitlab pipeline.
This is my Dockerfile: https://hastebin.com/share/ifirijonaj.bash but for TLDR reasons: it creates an image from python:3.12; I add username, password and certificate location as build arguments to Docker; I copy the certificate to a certificate location and install it; finally I run the above command.
Then, I build the docker image like this:
docker build -t image_name --build-arg "PRIVATE_PYPI_USERNAME=username" --build-arg "PRIVATE_PYPI_PASSWORD=password" --build-arg "CA_CERTIFICATE_LOCATION=cert.crt" --progress=plain .
My Dockerfile outputs the ${INDEX_URL} that it builds correctly, but the pip install command outputs this error:
#20 [16/19] RUN pip install -r requirements.txt --index-url "https://username:password@private.gitlab.internal/api/v4/projects/9/packages/pypi/simple/" --cert /usr/local/share/ca-certificates/cert.crt --trusted-host private.gitlab.internal
#20 0.678 Using pip 24.0 from /usr/local/lib/python3.12/site-packages/pip (python 3.12)
#20 0.728 Looking in indexes: https://username:****@private.gitlab.internal/api/v4/projects/9/packages/pypi/simple/
#20 1.034 ERROR: Could not find a version that satisfies the requirement black==24.4.0 (from versions: none)
#20 1.035 ERROR: No matching distribution found for black==24.4.0
#20 ERROR: process "/bin/sh -c pip install -r requirements.txt --index-url \"${INDEX_URL}\" --cert /usr/local/share/ca-certificates/cert.crt --trusted-host private.gitlab.internal" did not complete successfully: exit code: 1
------
> [16/19] RUN pip install -v -r requirements.txt --index-url "https://username:password@private.gitlab.internal/api/v4/projects/9/packages/pypi/simple/" --cert /usr/local/share/ca-certificates/cert.crt --trusted-host private.gitlab.internal:
0.678 Using pip 24.0 from /usr/local/lib/python3.12/site-packages/pip (python 3.12)
0.728 Looking in indexes: https://username:****@private.gitlab.internal/api/v4/projects/9/packages/pypi/simple/
1.034 ERROR: Could not find a version that satisfies the requirement black==24.4.0 (from versions: none)
1.035 ERROR: No matching distribution found for black==24.4.0
------
Dockerfile:31
--------------------
29 | # Install
30 | RUN pip install --upgrade pip
31 | >>> RUN pip install -v -r requirements.txt --index-url "${INDEX_URL}" --cert /usr/local/share/ca-certificates/cert.crt --trusted-host private.gitlab.internal
32 |
33 | # Make sure environment variables can be read
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -v -r requirements.txt --index-url \"${INDEX_URL}\" --cert /usr/local/share/ca-certificates/cert.crt --trusted-host private.gitlab.internal" did not complete successfully: exit code: 1
So it tries to install black==24.4.0 which is the first dependency in requirements.txt and (given that it works locally) exists in my private PyPI.
What could be the reason that the Docker build process can't pip install?
I already tried:
executing the command locally exactly as the docker build process outputs it
using curl to access black with the URL to the wheel from the container (using `--cacert`), which works correctly
different permutations of --no-cache-dir, --trusted-host, --no-deps, --no-index and quotes around ${INDEX_URL} but none of them seem to work
I solved my own problem. The issue was that the package registry didn't contain the wheels for the correct platform, only the ones that I downloaded locally (on Windows) and then uploaded, whereas the packages I needed should have been built for Linux.
Adding --platform manylinux2014_x86_64 --platform manylinux1_x86_64 --platform any to my pip download command and then uploading all wheels to the package registry solved the issue.