Pass Gitlab CI_JOB_TOKEN from Cloud Build to Dockerfile

I am in dire need of help on this. I feel like I've tried every piece of advice online to no avail. I must be missing something!!

Background

• I want to use Gitlab CICD and GCP Cloud Build to build a Docker image of a Python script.
• The Python script's dependencies are handled by Poetry.
• One of the dependencies is a Python package in a private repo in Gitlab's Package Registry.
• I need to pass the $CI_JOB_TOKEN environment variable from Gitlab CICD, through Cloud Build, and into Docker so that Poetry can install the Python package from the private repo.

I have tried hardcoding credentials into the Dockerfile and everything works as expected. It is only when trying to use the $CI_JOB_TOKEN variable that things fall apart!

.gitlab-ci.yml

...

.build:
  image:
    name: google/cloud-sdk:latest
  script:
    - gcloud builds submit --config=cloudbuild.yaml --substitutions=_CI_JOB_TOKEN="$CI_JOB_TOKEN"

...

cloudbuild.yaml

steps:
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: bash
    args:
      - -c
      - |
         DOCKER_BUILDKIT=1 docker build -t target --build arg=CI_JOB_TOKEN=${_CI_JOB_TOKEN} .

substitutions:
  _CI_JOB_TOKEN: replace at runtime

Dockerfile

FROM python:3.11.5-slim as python-base

#Install Poetry

RUN --mount-type=cache,target=/root/.cache \
  curl -sSL https://install.python-poetry.org | python -

WORKDIR /app
COPY poetry.lock pyproject.toml ./
COPY python_scripts/ python_scripts/

#Authenticate with Gitlab Package Registry with CI_JOB_TOKEN
ARG CI_JOB_TOKEN

RUN poetry config http-basic.gitlab gitlab-ci-token $CI_JOB_TOKEN

#Install dependencies
RUN --mount=type=cache,target=/root/.cache \
  poetry install --no-root --only-main

Pipeline Error

#19 7.513   RuntimeError
#19 7.513 
#19 7.513   Unable to find installation candidates for package_name (0.1.4)
#19 7.513 
#19 7.513   at /opt/poetry/venv/lib/python3.11/site-packages/poetry/installation/chooser.py:74 in choose_for
#19 7.532        70│ 
#19 7.532        71│             links.append(link)
#19 7.532        72│ 
#19 7.532        73│         if not links:
#19 7.532     →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
#19 7.533        75│ 
#19 7.533        76│         # Get the best link
#19 7.533        77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
#19 7.533        78│ 
#19 7.533 
#19 7.533 Cannot install package_name.
#19 7.533
#19 ERROR: executor failed running [/bin/sh -c poetry install --no-root --only main]: exit code: 1
------
 > [builder-base 8/8] RUN --mount=type=cache,target=/root/.cache     poetry install --no-root --only main:
------
executor failed running [/bin/sh -c poetry install --no-root --only main]: exit code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

Answering my own question.

My pipeline was running in repo_a. My package that I wanted to pull into my Docker image was in repo_b's Package Registry.

I was trying to authenticate with repo_a's $CI_JOB_TOKEN to repo_b's Package Registry but it kept failing. It failed because repo_a's $CI_JOB_TOKEN did not have access to authenticate with repo_b!!

I fixed the issue by creating a Deploy Token for repo_b and passing it to Docker in repo_a's pipeline.