docker google-cloud-platform google-cloud-build docker-buildkit cloudbuild.yaml

Cloud Build fails to use Docker BuildKit Cache

We are using Cloud Build to deploy our Docker container to Cloud Run. We would like to speed up the process with caching, as described here:

https://cloud.google.com/build/docs/optimize-builds/speeding-up-builds#using_a_cached_docker_image

Despite adding the --cache-from argument, no caching occurs and build times are consistently long.

Here is our current cloudbuild.yaml.

steps:
  # Pull the cached docker image
  - name: "gcr.io/cloud-builders/docker"
    entrypoint: "bash"
    args:
      [
        "-c",
        "docker pull us-west1-docker.pkg.dev/$PROJECT_ID/app-backend/app:latest",
      ]

  # Build the new docker image
  - name: "gcr.io/cloud-builders/docker"
    env:
      - DOCKER_BUILDKIT=1
    args:
      [
        "build",
        "-t",
        "us-west1-docker.pkg.dev/$PROJECT_ID/app-backend/app:latest",
        "--cache-from",
        "us-west1-docker.pkg.dev/$PROJECT_ID/app-backend/app:latest",
        "-f",
        "Dockerfile",
        ".",
      ]

  # Push to Artifact Registry
  - name: "gcr.io/cloud-builders/docker"
    args: ["push", "us-west1-docker.pkg.dev/$PROJECT_ID/app-backend/app:latest"]

  # Deploy to Cloud Run
  ...


  images: ["us-west1-docker.pkg.dev/$PROJECT_ID/app-backend/app:latest"]

Here is our Dockerfile:

# Inspired by 
# - https://github.com/Swiple/swiple/blob/main/backend/Dockerfile
# - https://github.com/fastapi/full-stack-fastapi-template/blob/master/backend/Dockerfile
###############################################
# Base Image
###############################################
FROM python:3.12-slim AS python-base

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    PYSETUP_PATH="/code" \
    VENV_PATH="/code/.venv"

###############################################
# Builder Image
###############################################
FROM python-base AS builder-base

WORKDIR $PYSETUP_PATH

# Install uv
COPY --from=ghcr.io/astral-sh/uv:0.5.11 /uv /uvx /bin/

# Compile bytecode
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#compiling-bytecode
ENV UV_COMPILE_BYTECODE=1

# uv Cache
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#caching
ENV UV_LINK_MODE=copy

# Install dependencies
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#intermediate-layers
RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --frozen --no-install-project

###############################################
# Production Image
###############################################
FROM python-base AS production

# https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#debian-11
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
    libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz-subset0 libjpeg-dev libopenjp2-7-dev libffi-dev && \
    apt clean && \
    rm -rf /var/cache/apt/*


# Place executables in the environment at the front of the path
# Ref: https://docs.astral.sh/uv/guides/integration/docker/#using-the-environment
ENV PATH="$VENV_PATH/bin:$PATH"

COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH

# Copy the app
COPY app $PYSETUP_PATH/app

# Change to the directory where the app is
WORKDIR $PYSETUP_PATH/app

# Copy entrypoint script
COPY entrypoint.sh /entrypoint.sh

# Make entrypoint script executable
RUN chmod +x /entrypoint.sh

# Use the entrypoint script
ENTRYPOINT ["/entrypoint.sh"]

And here are our Cloud Build Logs:

Creating temporary archive of 250 file(s) totalling 951.5 KiB before compression.
Some files were not included in the source upload.

Check the gcloud log [/home/runner/.config/gcloud/logs/2025.05.09/21.42.09.122982.log] to see which files and the contents of the
default gcloudignore file used (see `$ gcloud topic gcloudignore` to learn
more).

Uploading tarball of [.] to [gs://staging_cloudbuild/source/123456.047118-255aa2e8c0724ec9ba981757619f0b11.tgz]
Created [https://cloudbuild.googleapis.com/v1/projects/staging/locations/global/builds/a0cefb35-48c2-4ff4-8846-aebc16e948d0].
Logs are available at [ https://console.cloud.google.com/cloud-build/builds/a0cefb35-48c2-4ff4-8846-aebc16e948d0?project=1234 ].
Waiting for build to complete. Polling interval: 1 second(s).
----------------------------- REMOTE BUILD OUTPUT ------------------------------
starting build "a0cefb35-48c2-4ff4-8846-aebc16e948d0"

FETCHSOURCE
Fetching storage object: gs://staging_cloudbuild/source/123456.047118-255aa2e8c0724ec9ba981757619f0b11.tgz#123456683618
Copying gs://staging_cloudbuild/source/123456.047118-255aa2e8c0724ec9ba981757619f0b11.tgz#123456683618...
/ [0 files][    0.0 B/204.4 KiB]                                                
/ [1 files][204.4 KiB/204.4 KiB]                                                
Operation completed over 1 objects/204.4 KiB.
BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/docker
Step #0: latest: Pulling from staging/backend/my-project
Step #0: 61ffccc6e275: Pulling fs layer
Step #0: 1454c05bd781: Pulling fs layer
Step #0: 7b75476b4a2f: Pulling fs layer
Step #0: fb7b924ac822: Pulling fs layer
Step #0: 8c15955c9551: Pulling fs layer
Step #0: c5bf81559fe6: Pulling fs layer
Step #0: ad78a099a4a9: Pulling fs layer
Step #0: 4f4fb700ef54: Pulling fs layer
Step #0: 50a921cb8028: Pulling fs layer
Step #0: 252bbfc132e7: Pulling fs layer
Step #0: 8c15955c9551: Waiting
Step #0: c5bf81559fe6: Waiting
Step #0: ad78a099a4a9: Waiting
Step #0: 4f4fb700ef54: Waiting
Step #0: 50a921cb8028: Waiting
Step #0: 252bbfc132e7: Waiting
Step #0: fb7b924ac822: Waiting
Step #0: 1454c05bd781: Verifying Checksum
Step #0: 1454c05bd781: Download complete
Step #0: 61ffccc6e275: Verifying Checksum
Step #0: 61ffccc6e275: Download complete
Step #0: 7b75476b4a2f: Download complete
Step #0: fb7b924ac822: Verifying Checksum
Step #0: fb7b924ac822: Download complete
Step #0: ad78a099a4a9: Verifying Checksum
Step #0: ad78a099a4a9: Download complete
Step #0: 61ffccc6e275: Pull complete
Step #0: 8c15955c9551: Verifying Checksum
Step #0: 8c15955c9551: Download complete
Step #0: 4f4fb700ef54: Verifying Checksum
Step #0: 4f4fb700ef54: Download complete
Step #0: 1454c05bd781: Pull complete
Step #0: 50a921cb8028: Verifying Checksum
Step #0: 50a921cb8028: Download complete
Step #0: 252bbfc132e7: Verifying Checksum
Step #0: 252bbfc132e7: Download complete
Step #0: c5bf81559fe6: Verifying Checksum
Step #0: c5bf81559fe6: Download complete
Step #0: 7b75476b4a2f: Pull complete
Step #0: fb7b924ac822: Pull complete
Step #0: 8c15955c9551: Pull complete
Step #0: c5bf81559fe6: Pull complete
Step #0: ad78a099a4a9: Pull complete
Step #0: 4f4fb700ef54: Pull complete
Step #0: 50a921cb8028: Pull complete
Step #0: 252bbfc132e7: Pull complete
Step #0: Digest: sha256:399f3eb2385079ad2b18c85a50e99f15bc952de7463be963716f33aa3a8ab14e
Step #0: Status: Downloaded newer image for us-west1-docker.pkg.dev/staging/backend/my-project:latest
Step #0: us-west1-docker.pkg.dev/staging/backend/my-project:latest
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: #1 [internal] load build definition from Dockerfile
Step #1: #1 sha256:9cbc4d893a9f4448686336aaa226d34861349b262300f6679b404dae7bfbebef
Step #1: #1 transferring dockerfile: 2.32kB done
Step #1: #1 DONE 0.0s
Step #1: 
Step #1: #2 [internal] load .dockerignore
Step #1: #2 sha256:f24ba53ac79432cde23657215efaf519a52d6fc6b536c2a211a7ff9b4dce9cf1
Step #1: #2 transferring context: 2B done
Step #1: #2 DONE 0.0s
Step #1: 
Step #1: #3 [internal] load metadata for docker.io/library/python:3.12-slim
Step #1: #3 sha256:282c60c0cd524c3474d0420c244e3d5df8cc4a2d2881ca24f552a4fde5d61527
Step #1: #3 DONE 1.5s
Step #1: 
Step #1: #16 importing cache manifest from us-west1-docker.pkg.dev/staging/backend/my-project:latest
Step #1: #16 sha256:1ff226454f469d04631fad50c6b8166eba345d361bf15fdcdbf5882cff1ab9ad
Step #1: #16 DONE 0.0s
Step #1: 
Step #1: #4 [python-base 1/1] FROM docker.io/library/python:3.12-slim@sha256:bae1a061b657f403aaacb1069a7f67d91f7ef5725ab17ca36abc5f1b2797ff92
Step #1: #4 sha256:b14713a920f249ec49bc634786fa9c3859971f049120f74f3de2d54e70a41081
Step #1: #4 resolve docker.io/library/python:3.12-slim@sha256:bae1a061b657f403aaacb1069a7f67d91f7ef5725ab17ca36abc5f1b2797ff92 done
Step #1: #4 sha256:1764f1feebd43be702ce5c216144deb0a78d59bbcf4b9a1cb3455eda7e98445b 5.51kB / 5.51kB done
Step #1: #4 sha256:bae1a061b657f403aaacb1069a7f67d91f7ef5725ab17ca36abc5f1b2797ff92 9.13kB / 9.13kB done
Step #1: #4 sha256:9a8248e1cb202bc376cda8f5ff11c96239ef3ba32c80288caf71007d007d9a95 1.75kB / 1.75kB done
Step #1: #4 DONE 0.1s
Step #1: 
Step #1: #8 [internal] load build context
Step #1: #8 sha256:47e66752fe090320229b71563a1e2c4b28c973bf705c9224d8193e1587793f4e
Step #1: #8 transferring context: 977.76kB 0.1s done
Step #1: #8 DONE 0.1s
Step #1: 
Step #1: #6 [builder-base 1/3] WORKDIR /code
Step #1: #6 sha256:97d1f4576c7b2413f902ddca2a1ddd4e06fb83da5081c50d161c3c4d8d7d0c1d
Step #1: #6 DONE 0.1s
Step #1: 
Step #1: #5 [production 1/6] RUN apt-get update     && apt-get install --no-install-recommends -y     libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz-subset0 libjpeg-dev libopenjp2-7-dev libffi-dev &&     apt clean &&     rm -rf /var/cache/apt/*
Step #1: #5 sha256:127f876894721642d8e68620e446ec383db0da1f8c5700cc63a110025848769a
Step #1: #5 0.533 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
Step #1: #5 0.554 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
Step #1: #5 0.554 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Step #1: #5 0.710 Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8792 kB]
Step #1: #5 0.920 Get:5 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [512 B]
Step #1: #5 1.074 Get:6 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [258 kB]
Step #1: #5 ...
Step #1: 
Step #1: #15 FROM ghcr.io/astral-sh/uv:0.5.11
Step #1: #15 sha256:afa619e618354c508c37fce0e6519fb9ea70d64716d685814d59618f73caca7d
Step #1: #15 resolve ghcr.io/astral-sh/uv:0.5.11 0.5s done
Step #1: #15 sha256:0ac957607303916420297a4c9c213bb33fbd3c888f9cd7f4f7273596ebf42b85 2.19kB / 2.19kB done
Step #1: #15 sha256:7e479fa39802632c25b4e5c14ddfab9c5f443cd7c89626a0408d31a0b7afc193 669B / 669B done
Step #1: #15 sha256:19979093e3da84cd28f7b6f852513b795e084427176ef9fbfac58642e69b461a 1.30kB / 1.30kB done
Step #1: #15 sha256:02952ea2db8844012cecc744babe0000d0784fc901ee9ea6fcd6157d64c0db30 14.91MB / 14.91MB 0.2s done
Step #1: #15 sha256:9309c81c6be946e5798e9ac53d13cbdbc17666b1c9b728ce18e27d482d928acc 94B / 94B 0.1s done
Step #1: #15 extracting sha256:02952ea2db8844012cecc744babe0000d0784fc901ee9ea6fcd6157d64c0db30 1.1s done
Step #1: #15 extracting sha256:9309c81c6be946e5798e9ac53d13cbdbc17666b1c9b728ce18e27d482d928acc done
Step #1: #15 DONE 1.4s
Step #1: 
Step #1: #5 [production 1/6] RUN apt-get update     && apt-get install --no-install-recommends -y     libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz-subset0 libjpeg-dev libopenjp2-7-dev libffi-dev &&     apt clean &&     rm -rf /var/cache/apt/*
Step #1: #5 sha256:127f876894721642d8e68620e446ec383db0da1f8c5700cc63a110025848769a
Step #1: #5 ...
Step #1: 
Step #1: #7 [builder-base 2/3] COPY --from=ghcr.io/astral-sh/uv:0.5.11 /uv /uvx /bin/
Step #1: #7 sha256:739db55ccc6aa6a672e9f0134b713791db5f81de687a1b93fe4fb116f6a1dc63
Step #1: #7 DONE 0.2s
Step #1: 
Step #1: #9 [builder-base 3/3] RUN --mount=type=cache,target=/root/.cache/uv     --mount=type=bind,source=uv.lock,target=uv.lock     --mount=type=bind,source=pyproject.toml,target=pyproject.toml     uv sync --frozen --no-install-project
Step #1: #9 sha256:c8a181ccec22848154f4d393e15efcda0e5351dd4a479f3d422de27245b031e5
Step #1: #9 0.779 Using CPython 3.12.10 interpreter at: /usr/local/bin/python3
Step #1: #9 0.779 Creating virtual environment at: .venv
Step #1: #9 ...
Step #1: 
Step #1: #5 [production 1/6] RUN apt-get update     && apt-get install --no-install-recommends -y     libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz-subset0 libjpeg-dev libopenjp2-7-dev libffi-dev &&     apt clean &&     rm -rf /var/cache/apt/*
Step #1: #5 sha256:127f876894721642d8e68620e446ec383db0da1f8c5700cc63a110025848769a
Step #1: #5 2.893 Fetched 9305 kB in 2s (3898 kB/s)
Step #1: #5 2.893 Reading package lists...
Step #1: #5 4.007 Reading package lists...
Step #1: #5 5.226 Building dependency tree...
Step #1: #5 5.495 Reading state information...
Step #1: #5 5.834 The following additional packages will be installed:
Step #1: #5 5.835   fontconfig fontconfig-config fonts-dejavu-core libbrotli1 libc-dev-bin
Step #1: #5 5.836   libc6-dev libcrypt-dev libdatrie1 libexpat1 libfontconfig1 libfreetype6
Step #1: #5 5.836   libfribidi0 libglib2.0-0 libgraphite2-3 libharfbuzz0b libjpeg62-turbo
Step #1: #5 5.837   libjpeg62-turbo-dev libnsl-dev libopenjp2-7 libpng16-16 libthai-data
Step #1: #5 5.839   libthai0 libtirpc-dev linux-libc-dev rpcsvc-proto
Step #1: #5 5.842 Suggested packages:
Step #1: #5 5.842   glibc-doc manpages-dev low-memory-monitor
Step #1: #5 5.842 Recommended packages:
Step #1: #5 5.842   manpages manpages-dev libc-devtools libglib2.0-data shared-mime-info
Step #1: #5 5.842   xdg-user-dirs
Step #1: #5 6.133 The following NEW packages will be installed:
Step #1: #5 6.134   fontconfig fontconfig-config fonts-dejavu-core libbrotli1 libc-dev-bin
Step #1: #5 6.134   libc6-dev libcrypt-dev libdatrie1 libexpat1 libffi-dev libfontconfig1
Step #1: #5 6.134   libfreetype6 libfribidi0 libglib2.0-0 libgraphite2-3 libharfbuzz-subset0
Step #1: #5 6.135   libharfbuzz0b libjpeg-dev libjpeg62-turbo libjpeg62-turbo-dev libnsl-dev
Step #1: #5 6.135   libopenjp2-7 libopenjp2-7-dev libpango-1.0-0 libpangoft2-1.0-0 libpng16-16
Step #1: #5 6.136   libthai-data libthai0 libtirpc-dev linux-libc-dev rpcsvc-proto
Step #1: #5 6.171 0 upgraded, 31 newly installed, 0 to remove and 0 not upgraded.
Step #1: #5 6.171 Need to get 14.8 MB of archives.
Step #1: #5 6.171 After this operation, 43.6 MB of additional disk space will be used.
Step #1: #5 6.171 Get:1 http://deb.debian.org/debian bookworm/main amd64 libexpat1 amd64 2.5.0-1+deb12u1 [98.9 kB]
Step #1: #5 6.178 Get:2 http://deb.debian.org/debian bookworm/main amd64 libbrotli1 amd64 1.0.9-2+b6 [275 kB]
Step #1: #5 6.183 Get:3 http://deb.debian.org/debian bookworm/main amd64 libpng16-16 amd64 1.6.39-2 [276 kB]
Step #1: #5 6.187 Get:4 http://deb.debian.org/debian-security bookworm-security/main amd64 libfreetype6 amd64 2.12.1+dfsg-5+deb12u4 [398 kB]
Step #1: #5 6.191 Get:5 http://deb.debian.org/debian bookworm/main amd64 fonts-dejavu-core all 2.37-6 [1068 kB]
Step #1: #5 6.203 Get:6 http://deb.debian.org/debian bookworm/main amd64 fontconfig-config amd64 2.14.1-4 [315 kB]
Step #1: #5 6.208 Get:7 http://deb.debian.org/debian bookworm/main amd64 libfontconfig1 amd64 2.14.1-4 [386 kB]
Step #1: #5 6.213 Get:8 http://deb.debian.org/debian bookworm/main amd64 fontconfig amd64 2.14.1-4 [449 kB]
Step #1: #5 6.219 Get:9 http://deb.debian.org/debian bookworm/main amd64 libc-dev-bin amd64 2.36-9+deb12u10 [47.1 kB]
Step #1: #5 6.221 Get:10 http://deb.debian.org/debian-security bookworm-security/main amd64 linux-libc-dev amd64 6.1.135-1 [2136 kB]
Step #1: #5 6.241 Get:11 http://deb.debian.org/debian bookworm/main amd64 libcrypt-dev amd64 1:4.4.33-2 [118 kB]
Step #1: #5 6.243 Get:12 http://deb.debian.org/debian bookworm/main amd64 libtirpc-dev amd64 1.3.3+ds-1 [191 kB]
Step #1: #5 6.246 Get:13 http://deb.debian.org/debian bookworm/main amd64 libnsl-dev amd64 1.3.0-2 [66.4 kB]
Step #1: #5 6.249 Get:14 http://deb.debian.org/debian bookworm/main amd64 rpcsvc-proto amd64 1.4.3-1 [63.3 kB]
Step #1: #5 6.250 Get:15 http://deb.debian.org/debian bookworm/main amd64 libc6-dev amd64 2.36-9+deb12u10 [1903 kB]
Step #1: #5 6.274 Get:16 http://deb.debian.org/debian bookworm/main amd64 libdatrie1 amd64 0.2.13-2+b1 [43.3 kB]
Step #1: #5 6.275 Get:17 http://deb.debian.org/debian bookworm/main amd64 libffi-dev amd64 3.4.4-1 [59.4 kB]
Step #1: #5 6.277 Get:18 http://deb.debian.org/debian bookworm/main amd64 libfribidi0 amd64 1.0.8-2.1 [65.0 kB]
Step #1: #5 6.279 Get:19 http://deb.debian.org/debian bookworm/main amd64 libglib2.0-0 amd64 2.74.6-2+deb12u5 [1403 kB]
Step #1: #5 6.295 Get:20 http://deb.debian.org/debian bookworm/main amd64 libgraphite2-3 amd64 1.3.14-1 [81.2 kB]
Step #1: #5 6.297 Get:21 http://deb.debian.org/debian bookworm/main amd64 libharfbuzz0b amd64 6.0.0+dfsg-3 [1945 kB]
Step #1: #5 6.319 Get:22 http://deb.debian.org/debian bookworm/main amd64 libharfbuzz-subset0 amd64 6.0.0+dfsg-3 [1931 kB]
Step #1: #5 6.340 Get:23 http://deb.debian.org/debian bookworm/main amd64 libjpeg62-turbo amd64 1:2.1.5-2 [166 kB]
Step #1: #5 6.342 Get:24 http://deb.debian.org/debian bookworm/main amd64 libjpeg62-turbo-dev amd64 1:2.1.5-2 [290 kB]
Step #1: #5 6.347 Get:25 http://deb.debian.org/debian bookworm/main amd64 libjpeg-dev amd64 1:2.1.5-2 [71.8 kB]
Step #1: #5 6.349 Get:26 http://deb.debian.org/debian bookworm/main amd64 libopenjp2-7 amd64 2.5.0-2+deb12u1 [189 kB]
Step #1: #5 6.352 Get:27 http://deb.debian.org/debian bookworm/main amd64 libopenjp2-7-dev amd64 2.5.0-2+deb12u1 [263 kB]
Step #1: #5 6.355 Get:28 http://deb.debian.org/debian bookworm/main amd64 libthai-data all 0.1.29-1 [176 kB]
Step #1: #5 6.358 Get:29 http://deb.debian.org/debian bookworm/main amd64 libthai0 amd64 0.1.29-1 [57.5 kB]
Step #1: #5 6.360 Get:30 http://deb.debian.org/debian bookworm/main amd64 libpango-1.0-0 amd64 1.50.12+ds-1 [212 kB]
Step #1: #5 6.363 Get:31 http://deb.debian.org/debian bookworm/main amd64 libpangoft2-1.0-0 amd64 1.50.12+ds-1 [47.4 kB]
Step #1: #5 6.724 debconf: delaying package configuration, since apt-utils is not installed
Step #1: #5 6.795 Fetched 14.8 MB in 0s (69.0 MB/s)
Step #1: #5 6.845 Selecting previously unselected package libexpat1:amd64.
Step #1: #5 6.846 (Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 6688 files and directories currently installed.)
Step #1: #5 6.905 Preparing to unpack .../00-libexpat1_2.5.0-1+deb12u1_amd64.deb ...
Step #1: #5 6.914 Unpacking libexpat1:amd64 (2.5.0-1+deb12u1) ...
Step #1: #5 6.984 Selecting previously unselected package libbrotli1:amd64.
Step #1: #5 6.987 Preparing to unpack .../01-libbrotli1_1.0.9-2+b6_amd64.deb ...
Step #1: #5 6.990 Unpacking libbrotli1:amd64 (1.0.9-2+b6) ...
Step #1: #5 7.092 Selecting previously unselected package libpng16-16:amd64.
Step #1: #5 7.095 Preparing to unpack .../02-libpng16-16_1.6.39-2_amd64.deb ...
Step #1: #5 7.099 Unpacking libpng16-16:amd64 (1.6.39-2) ...
Step #1: #5 7.179 Selecting previously unselected package libfreetype6:amd64.
Step #1: #5 7.181 Preparing to unpack .../03-libfreetype6_2.12.1+dfsg-5+deb12u4_amd64.deb ...
Step #1: #5 7.186 Unpacking libfreetype6:amd64 (2.12.1+dfsg-5+deb12u4) ...
Step #1: #5 7.310 Selecting previously unselected package fonts-dejavu-core.
Step #1: #5 7.313 Preparing to unpack .../04-fonts-dejavu-core_2.37-6_all.deb ...
(... more package installs)
Step #1: #5 10.58 Preparing to unpack .../29-libpango-1.0-0_1.50.12+ds-1_amd64.deb ...
Step #1: #5 10.58 Unpacking libpango-1.0-0:amd64 (1.50.12+ds-1) ...
Step #1: #5 10.65 Selecting previously unselected package libpangoft2-1.0-0:amd64.
Step #1: #5 10.66 Preparing to unpack .../30-libpangoft2-1.0-0_1.50.12+ds-1_amd64.deb ...
Step #1: #5 10.66 Unpacking libpangoft2-1.0-0:amd64 (1.50.12+ds-1) ...
Step #1: #5 10.72 Setting up libexpat1:amd64 (2.5.0-1+deb12u1) ...
(...more setting up)
Step #1: #5 10.80 Setting up libtirpc-dev:amd64 (1.3.3+ds-1) ...
Step #1: #5 10.80 Setting up rpcsvc-proto (1.4.3-1) ...
Step #1: #5 10.81 Setting up libjpeg62-turbo:amd64 (1:2.1.5-2) ...
Step #1: #5 10.82 Setting up libfribidi0:amd64 (1.0.8-2.1) ...
Step #1: #5 10.83 Setting up libpng16-16:amd64 (1.6.39-2) ...
Step #1: #5 10.84 Setting up fonts-dejavu-core (2.37-6) ...
Step #1: #5 10.88 Setting up libnsl-dev:amd64 (1.3.0-2) ...
Step #1: #5 10.89 Setting up libcrypt-dev:amd64 (1:4.4.33-2) ...
Step #1: #5 10.91 Setting up libopenjp2-7:amd64 (2.5.0-2+deb12u1) ...
Step #1: #5 10.92 Setting up libthai-data (0.1.29-1) ...
Step #1: #5 10.93 Setting up libc-dev-bin (2.36-9+deb12u10) ...
Step #1: #5 10.94 Setting up fontconfig-config (2.14.1-4) ...
Step #1: #5 11.09 debconf: unable to initialize frontend: Dialog
Step #1: #5 11.09 debconf: (TERM is not set, so the dialog frontend is not usable.)
Step #1: #5 11.09 debconf: falling back to frontend: Readline
Step #1: #5 11.10 debconf: unable to initialize frontend: Readline
Step #1: #5 11.10 debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
Step #1: #5 11.10 debconf: falling back to frontend: Teletype
Step #1: #5 11.37 Setting up libthai0:amd64 (0.1.29-1) ...
Step #1: #5 11.38 Setting up libfreetype6:amd64 (2.12.1+dfsg-5+deb12u4) ...
Step #1: #5 11.39 Setting up libopenjp2-7-dev:amd64 (2.5.0-2+deb12u1) ...
Step #1: #5 11.40 Setting up libc6-dev:amd64 (2.36-9+deb12u10) ...
Step #1: #5 11.40 Setting up libharfbuzz0b:amd64 (6.0.0+dfsg-3) ...
Step #1: #5 11.42 Setting up libfontconfig1:amd64 (2.14.1-4) ...
Step #1: #5 11.43 Setting up fontconfig (2.14.1-4) ...
Step #1: #5 11.43 Regenerating fonts cache... done.
Step #1: #5 13.47 Setting up libpango-1.0-0:amd64 (1.50.12+ds-1) ...
Step #1: #5 13.47 Setting up libjpeg62-turbo-dev:amd64 (1:2.1.5-2) ...
Step #1: #5 13.48 Setting up libharfbuzz-subset0:amd64 (6.0.0+dfsg-3) ...
Step #1: #5 13.49 Setting up libpangoft2-1.0-0:amd64 (1.50.12+ds-1) ...
Step #1: #5 13.50 Setting up libjpeg-dev:amd64 (1:2.1.5-2) ...
Step #1: #5 13.50 Processing triggers for libc-bin (2.36-9+deb12u10) ...
Step #1: #5 13.60 
Step #1: #5 13.61 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Step #1: #5 13.61
Step #1: #5 DONE 16.6s
Step #1: 
Step #1: #9 [builder-base 3/3] RUN --mount=type=cache,target=/root/.cache/uv     --mount=type=bind,source=uv.lock,target=uv.lock     --mount=type=bind,source=pyproject.toml,target=pyproject.toml     uv sync --frozen --no-install-project
Step #1: #9 sha256:c8a181ccec22848154f4d393e15efcda0e5351dd4a479f3d422de27245b031e5
Step #1: #9 9.573 Prepared 112 packages in 8.74s
Step #1: #9 10.58 Installed 112 packages in 1.00s
Step #1: #9 26.98 Bytecode compiled 5440 files in 16.39s
Step #1: #9 26.98  + alembic==1.15.2
Step #1: #9 26.98  + annotated-types==0.7.0
Step #1: #9 26.98  + anyio==4.9.0
Step #1: #9 26.98  + astroid==3.3.9
Step #1: #9 26.98  + asyncpg==0.30.0
Step #1: #9 26.98  + brotli==1.1.0
... (other packages)
Step #1: #9 27.00  + watchfiles==1.0.5
Step #1: #9 27.00  + weasyprint==65.1
Step #1: #9 27.00  + webencodings==0.5.1
Step #1: #9 27.00  + websockets==15.0.1
Step #1: #9 27.00  + zopfli==0.2.3.post1
Step #1: #9 DONE 27.4s
Step #1: 
Step #1: #10 [production 2/6] COPY --from=builder-base /code /code
Step #1: #10 sha256:1bac80cef93551eb1b79742c86c52bc939358706782d12c262a6b2b59cd0e6a0
Step #1: #10 DONE 4.0s
Step #1: 
Step #1: #11 [production 3/6] COPY app /code/app
Step #1: #11 sha256:1b2b75d74591e66a9ab4935e541a8827c1b0d5b6e507b8d85ac5fd9e68caf8a4
Step #1: #11 DONE 0.1s
Step #1: 
Step #1: #12 [production 4/6] WORKDIR /code/app
Step #1: #12 sha256:05d0c99cddf8edadc974d30481e8f5f4a6311715209bfd7fa830c4baaf944fd2
Step #1: #12 DONE 0.0s
Step #1: 
Step #1: #13 [production 5/6] COPY entrypoint.sh /entrypoint.sh
Step #1: #13 sha256:db9e751988ef935274e3429c922f02d3f985adcae2c39ca9e13af2c8fa70b2e2
Step #1: #13 DONE 0.0s
Step #1: 
Step #1: #14 [production 6/6] RUN chmod +x /entrypoint.sh
Step #1: #14 sha256:8c91bcfef26bf154e20db1bc3bb053a5a2ce69f7b48489cab90bd085c8f5f5c7
Step #1: #14 DONE 0.3s
Step #1: 
Step #1: #17 exporting to image
Step #1: #17 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
Step #1: #17 exporting layers
Step #1: #17 exporting layers 4.6s done
Step #1: #17 writing image sha256:99eb8b0e9cae8b807fb87a75827af983ae96595cebaae6fec312294c27b2cfb6 done
Step #1: #17 naming to us-west1-docker.pkg.dev/staging/backend/my-project:latest done
Step #1: #17 DONE 4.7s
Finished Step #1
Starting Step #2
Step #2: Already have image (with digest): gcr.io/cloud-builders/docker
Step #2: The push refers to repository [us-west1-docker.pkg.dev/staging/backend/my-project]
Step #2: 542a7cdd396d: Preparing
Step #2: 48188b1d6aa9: Preparing
Step #2: 5f70bf18a086: Preparing
Step #2: 9521682e6364: Preparing
Step #2: 1ad9fe87cd1a: Preparing
Step #2: 7b6de6133c5e: Preparing
Step #2: 84a86f614ade: Preparing
Step #2: 9aeba49f9ae6: Preparing
Step #2: d50d320ea099: Preparing
Step #2: 6c4c763d22d0: Preparing
Step #2: 7b6de6133c5e: Waiting
Step #2: 84a86f614ade: Waiting
Step #2: 9aeba49f9ae6: Waiting
Step #2: d50d320ea099: Waiting
Step #2: 6c4c763d22d0: Waiting
Step #2: 5f70bf18a086: Layer already exists
Step #2: 9521682e6364: Pushed
Step #2: 542a7cdd396d: Pushed
Step #2: 48188b1d6aa9: Pushed
Step #2: 84a86f614ade: Layer already exists
Step #2: 9aeba49f9ae6: Layer already exists
Step #2: d50d320ea099: Layer already exists
Step #2: 6c4c763d22d0: Layer already exists
Step #2: 7b6de6133c5e: Pushed
Step #2: 1ad9fe87cd1a: Pushed
Step #2: latest: digest: sha256:c22c7cda93c213a3172d3e5544b662ea48d4d497e029bf8e8a2734da290a0564 size: 2414
Finished Step #2
Starting Step #3
Step #3: Pulling image: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
Step #3: slim: Pulling from google.com/cloudsdktool/cloud-sdk
Step #3: cf05a52c0235: Already exists
Step #3: c4b0c8d90e26: Pulling fs layer
Step #3: d0c5bbb31fea: Pulling fs layer
Step #3: 8bd4cadf3b57: Pulling fs layer
Step #3: 9e1d07fddb81: Pulling fs layer
Step #3: 7e4318a06f9a: Pulling fs layer
Step #3: 9e1d07fddb81: Waiting
Step #3: 7e4318a06f9a: Waiting
Step #3: 8bd4cadf3b57: Verifying Checksum
Step #3: 8bd4cadf3b57: Download complete
Step #3: d0c5bbb31fea: Verifying Checksum
Step #3: d0c5bbb31fea: Download complete
Step #3: c4b0c8d90e26: Verifying Checksum
Step #3: c4b0c8d90e26: Download complete
Step #3: 7e4318a06f9a: Verifying Checksum
Step #3: 7e4318a06f9a: Download complete
Step #3: c4b0c8d90e26: Pull complete
Step #3: d0c5bbb31fea: Pull complete
Step #3: 8bd4cadf3b57: Pull complete
Step #3: 9e1d07fddb81: Download complete
Step #3: 9e1d07fddb81: Pull complete
Step #3: 7e4318a06f9a: Pull complete
Step #3: Digest: sha256:c1f6897d634f4bb9c0755e29fe9641e1c90f3b59588df65d91178b52611b6593
Step #3: Status: Downloaded newer image for gcr.io/google.com/cloudsdktool/cloud-sdk:slim
Step #3: gcr.io/google.com/cloudsdktool/cloud-sdk:slim

INFO: The service account running this build projects/staging/serviceAccounts/1234-compute@developer.gserviceaccount.com does not have permission to write logs to Cloud Logging. To fix this, grant the Logs Writer (roles/logging.logWriter) role to the service account.

1 message(s) issued.
Step #3: Deploying...
Step #3: Creating

What I’ve tried:
• Confirmed the image exists in Artifact Registry before the build. • Verified the docker pull step succeeds. • Added DOCKER_BUILDKIT=1 as some docs suggest this is needed. • Checked build logs for any sign of caching (e.g., “Using cache”) — none found.

What’s preventing Docker layer caching from working in this setup?

Is it due to BuildKit being enabled? What’s the correct way to enable caching in Cloud Build when deploying to Cloud Run using Docker BuildKit?

Any guidance would be appreciated!

Solution

What’s preventing Docker layer caching from working in this setup?

It's a combination of buildkit and multi-stage builds. Buildkit has additional flags to include cache metadata with the pushed image. And by default, only the layers of the final image are cached, not the previous stages of a multi-stage build.

To improve your command line, I'd recommend following Docker's guidance at https://docs.docker.com/build/cache/backends/

Specifically, you want to run a command similar to:

docker buildx build --push -t <registry>/<image> \
  --cache-to type=registry,ref=<registry>/<cache-image>,mode=max \
  --cache-from type=registry,ref=<registry>/<cache-image> .

Where the cache-image is separate from the built image, and the mode=max is used to include all stages.

Note that creating, storing, and retrieving the cache all adds overhead to a build, so you'll want to benchmark this overhead (vs without any caching) to ensure on average you are actually reducing the time for your builds.