After doing a lot of testing and moving things around I'm still running into the same issue as before. I'll try to keep this explanation short but there's a lot to unpack with this issue:
I have a Gitlab CI/CD pipeline that builds multi-arch docker images with buildx, pushes them to a staging folder in my Gitlab container registry, scans them for vulnerabilities, and then pushes that image to my dev and production registries when they meet my security compliance standards. Currently everything is working until I need to push the multi-arch image to my dev and production registries where I receive this error right when my bash script runs the docker push command:
####################################
Pushing Docker Manifest (alpine_ci:1243060338)...
docker push registry.gitlab.com/warpit/foundation/warp-ci/dev/alpine_ci:1243060338
The push refers to repository [registry.gitlab.com/warpit/foundation/warp-ci/dev/alpine_ci]
missing content: content digest sha256:20ca4eb37ffdd4345c836ba3e5c0c3df8fd8121cc9b6cc73910b8b52eac65e74: not found
Note: You're trying to push a manifest list/index which references multiple platform specific manifests, but not all of them are available locally or available to the remote repository.
Make sure you have all the referenced content and try again.
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1
Let me explain exactly what this push job is doing before moving forward:
My push CI job executes a bash script that logs into my container registry and pulls the multi-arch image that the previous build job created/pushed. After the push script pulls the multi-arch image, it then tags that image pointing it to the dev or production registry, and then pushes it.
I understand what this error is saying but I don't understand why this issue is occurring as I have my script run the docker manifest inspect command after it pulls the image so I can check it for debugging purposes. In the output below, you can see that the image digest the error is complaining about is present (Not going to show the whole output due to high verbosity):
####################################
Inspecting Docker Manifest (alpine_ci:1243060338)...
docker manifest inspect -v registry.gitlab.com/warpit/foundation/warp-ci/staging/alpine_ci:1243060338
{
"Ref": "registry.gitlab.com/warpit/foundation/warp-ci/staging/alpine_ci:1243060338@sha256:20ca4eb37ffdd4345c836ba3e5c0c3df8fd8121cc9b6cc73910b8b52eac65e74",
"Descriptor": {
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"digest": "sha256:20ca4eb37ffdd4345c836ba3e5c0c3df8fd8121cc9b6cc73910b8b52eac65e74",
"size": 840,
"platform": {
"architecture": "unknown",
"os": "unknown"
}
},
"Raw": "ewogICJzY2hlbWFWZXJzaW9uIjogMiwKICAibWVkaWFUeXBlIjogImFwcGxpY2F0aW9uL3ZuZC5vY2kuaW1hZ2UubWFuaWZlc3QudjEranNvbiIsCiAgImNvbmZpZyI6IHsKICAgICJtZWRpYVR5cGUiOiAiYXBwbGljYXRpb24vdm5kLm9jaS5pbWFnZS5jb25maWcudjEranNvbiIsCiAgICAiZGlnZXN0IjogInNoYTI1Njo1OWEyNzEyOGJmMGYxYjIxNmFkZmFjMmFkYjY3YzBhNjgzNzFiODdlZDcwODMyNTlhNDAzOWY2NjE5YmQyNDRiIiwKICAgICJzaXplIjogMjQxCiAgfSwKICAibGF5ZXJzIjogWwogICAgewogICAgICAibWVkaWFUeXBlIjogImFwcGxpY2F0aW9uL3ZuZC5pbi10b3RvK2pzb24iLAogICAgICAiZGlnZXN0IjogInNoYTI1NjoxMzc2Yzg1NjdkNDI2NWIwY2EzY2NkNDEwMmJiODA4NDQxZDdjYmM5ZWM2MGE4OWY0OTZlNmZiMjM1NzRjMzE3IiwKICAgICAgInNpemUiOiA1NzQ2ODc3LAogICAgICAiYW5ub3RhdGlvbnMiOiB7CiAgICAgICAgImluLXRvdG8uaW8vcHJlZGljYXRlLXR5cGUiOiAiaHR0cHM6Ly9zcGR4LmRldi9Eb2N1bWVudCIKICAgICAgfQogICAgfSwKICAgIHsKICAgICAgIm1lZGlhVHlwZSI6ICJhcHBsaWNhdGlvbi92bmQuaW4tdG90bytqc29uIiwKICAgICAgImRpZ2VzdCI6ICJzaGEyNTY6NjVmMDUzYmI2ZmQ3NmU3NjA0YjRiMGJhM2E3Mjc0N2UzNWY5N2UyNGFmYjAzMjkwNmNkNmFlZjgxODUxOGFkMyIsCiAgICAgICJzaXplIjogMTM1MywKICAgICAgImFubm90YXRpb25zIjogewogICAgICAgICJpbi10b3RvLmlvL3ByZWRpY2F0ZS10eXBlIjogImh0dHBzOi8vc2xzYS5kZXYvcHJvdmVuYW5jZS92MC4yIgogICAgICB9CiAgICB9CiAgXQp9",
"OCIManifest": {
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"digest": "sha256:59a27128bf0f1b216adfac2adb67c0a68371b87ed7083259a4039f6619bd244b",
"size": 241
},
"layers": [
{
"mediaType": "application/vnd.in-toto+json",
"digest": "sha256:1376c8567d4265b0ca3ccd4102bb808441d7cbc9ec60a89f496e6fb23574c317",
"size": 5746877,
"annotations": {
"in-toto.io/predicate-type": "https://spdx.dev/Document"
}
},
{
"mediaType": "application/vnd.in-toto+json",
"digest": "sha256:65f053bb6fd76e7604b4b0ba3a72747e35f97e24afb032906cd6aef818518ad3",
"size": 1353,
"annotations": {
"in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
}
}
]
}
}
]
I know 2 workaround solutions for this:
push script and push those instead.build job build architecture specific images for all architectures so that my push job can pull them and generate a manifest itself to push.Even though I know these will both fix my issue, that's not what I want to do. With that said, here are my questions:
One thing I haven't tested yet is using docker pull --platform in a for loop to pull the images for the specific architectures, pull the manifest itself, and then try to push from there. Once I have time to do so, I will update this ticket with my findings.
To whom it may concern,
I'm still partially confused as to why this issue exists in the first place considering the fact that when you pull a multi-arch image, it grabs all the hashes within it but your guess is as good as mine. Through a lot of testing I have found a reasonable solution that doesn't require me to re-build the images from scratch when trying to push them to my registries after running a docker build and SAST scan.
Now that I'm providing a solution that others may find useful, I will go into detail on how everything was setup and functions in my environment.
Recently I found that in order for you to be able to build/store multi-arch images locally, and capitalize on dockers built-in sbom and provenance attestations, you need to update docker engine to use the containerd image store. After enabling this and restarting the docker service on the Gitlab runner instance, I decided to update my pipeline, scripts, etc to support multi-arch images with sbom attestations.
With this in mind, I could now have my Gitlab runner build docker images of any architecture simultaneously, package them into a single manifest with their attestations, push it to my registry to be later scanned, and uploaded to my dev and production environments. In theory this was great and worked wonderfully until I had to pull the multi-arch image and push it to my registries after it was scanned for vulnerabilities.
After playing around a lot, I realized that the sbom and provenance attestations I was creating at build time, were being attached to the multi-arch image (manifest/JSON file) as the objects with the unknown architecture and OS. This was one of the things that was causing my problem. I don't fully understand why docker works this way but that's the way it is. After this, I changed my build command to:
docker buildx build --platform "linux/amd64,linux/arm64" --provenance false -t <image-tag> -f <path/to/dokcerfile> . --push
Once I did this, the unknown objects in my manifest were gone but we weren't out of the weeds yet as I was still getting errors saying that the arm64 hash couldn't be found. To fix this, I updated my pull command to a for loop that would iterate over all the architectures defined in the CI variable $TARGET_PLATFORMS:
#!/bin/bash
set -e
green='\033[0;32m'
blue='\033[0;34m'
clear='\033[0m'
###########################
# Script Requirnments
###########################
# This bash script requires the follwing environment variables to be set:
# - IMAGE_NAME
# - IMAGE_TAG
# - ENV
# - TARGET_PLATFORMS
###########################
# Login to Gitlab Registry
###########################
echo -e "${green}####################################${clear}"
echo -e "${blue}Logging into Gitlab Container Registry...${clear}"
docker login ${CI_REGISTRY} -u ${CI_REGISTRY_USER} -p ${CI_JOB_TOKEN}
echo -e "${green}####################################${clear}"
######################################
# Convert ${TARGET_PLATFORMS} to array
######################################
readarray -td ',' PLATFORMS < <(awk '{ gsub(/,[ ]+/,"\0"); print; }' <<<"$TARGET_PLATFORMS")
###########################
# Pull docker images
###########################
for platform in ${PLATFORMS[@]}
do
echo -e "${green}####################################${clear}"
echo -e "${blue}Pulling Docker Image (${IMAGE_NAME} | ${platform})...${clear}"
echo -e "${green}docker pull --platform ${platform} ${CI_REGISTRY_IMAGE}/staging/${IMAGE_NAME}:${IMAGE_TAG}${clear}"
docker pull --platform ${platform} ${CI_REGISTRY_IMAGE}/staging/${IMAGE_NAME}:${IMAGE_TAG}
echo -e "${green}####################################${clear}"
done
#################################
# Inspect Docker Manifest
#################################
echo -e "${green}####################################${clear}"
echo -e "${blue}Inspecting Docker Manifest (${IMAGE_NAME}:${IMAGE_TAG})...${clear}"
echo -e "${green}docker manifest inspect -v ${CI_REGISTRY_IMAGE}/staging/${IMAGE_NAME}:${IMAGE_TAG}${clear}"
docker manifest inspect -v ${CI_REGISTRY_IMAGE}/staging/${IMAGE_NAME}:${IMAGE_TAG}
echo -e "${green}####################################${clear}"
##############################
# Tag & Push Docker Manifest
##############################
echo -e "${green}####################################${clear}"
echo -e "${blue}Tagging Docker Manifest (${IMAGE_NAME}:${IMAGE_TAG})...${clear}"
echo -e "${green}docker tag ${CI_REGISTRY_IMAGE}/staging/${IMAGE_NAME}:${IMAGE_TAG} ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}:${IMAGE_TAG}${clear}"
docker tag ${CI_REGISTRY_IMAGE}/staging/${IMAGE_NAME}:${IMAGE_TAG} ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}:${IMAGE_TAG}
echo -e "${green}####################################${clear}"
echo -e "${blue}Pushing Docker Manifest (${IMAGE_NAME}:${IMAGE_TAG})...${clear}"
echo -e "${green}docker manifest push ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}:${IMAGE_TAG}${clear}"
docker push ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}:${IMAGE_TAG}
echo -e "${green}####################################${clear}"
##############################
# Tag & Push as latest
##############################
echo -e "${green}####################################${clear}"
echo -e "${blue}Tagging as latest (${IMAGE_NAME}:${IMAGE_TAG})...${clear}"
echo -e "${green}docker tag ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}:${IMAGE_TAG} ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}${clear}"
docker tag ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}:${IMAGE_TAG} ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}
echo -e "${green}####################################${clear}"
echo -e "${blue}Pushing as latest (${IMAGE_NAME}:${IMAGE_TAG})...${clear}"
echo -e "${green}docker manifest push ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}${clear}"
docker push ${CI_REGISTRY_IMAGE}/${ENV}/${IMAGE_NAME}
echo -e "${green}####################################${clear}"
After updating this, my push job finally succeeded without issues because all the images and their hashes were present when docker needed to push the manifest to it's new registry.
Sadly with these changes we lose the ability to use the docker built-in sbom and provenance attestations but you can easily create another stage/CI job to handle that for you after the image is built.
Furthermore, I'm not completely sure if pulling a multi-arch image really downloads the blueprint for all architectures of the image. My guess would be that it only pulls the blueprint for your devices specific architecture which is why I still had issues after removing the attestations.