GitLab CI/CD: building multiarch Docker images

I want an easy way to build multiarch Docker images in a GitLab runner. By easy, I mean that I just would have to add a .gitlab-ci.yml in my project and it would work.

Here is the .gitlab-ci.yml that I wrote. It builds a multiarch image using buildx and then pushes it to the GitLab registry:

image: cl00e9ment/buildx

services:
- name: docker:dind

variables:
  PLATFORMS: linux/amd64,linux/arm64
  TAG: latest

before_script:
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

build:
  stage: build
  script:
  - docker buildx build --platform "$PLATFORMS" -t "${CI_REGISTRY_IMAGE}:${TAG}" . --push

The problem is that the linux/arm64 platform isn't available.

Here is how I built the cl00e9ment/buildx image (strongly inspired from snadn/docker-buildx):

Here is the Dockerfile:

FROM docker:latest

ENV DOCKER_CLI_EXPERIMENTAL=enabled
ENV DOCKER_HOST=tcp://docker:2375/

RUN mkdir -p ~/.docker/cli-plugins \
  && wget -qO- https://api.github.com/repos/docker/buildx/releases/latest | grep "browser_download_url.*linux-amd64" | cut -d : -f 2,3 | tr -d '"' | xargs wget -O ~/.docker/cli-plugins/docker-buildx \
  && chmod a+x ~/.docker/cli-plugins/docker-buildx
RUN docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
RUN docker context create buildx \
    && docker buildx create buildx --name mybuilder \
    && docker buildx use mybuilder
RUN docker buildx inspect --bootstrap

...add here is the .gitlab-ci.yml file used to build and push the cl00e9ment/buildx image:

image: docker:latest

services:
  - name: docker:dind

before_script:
  - docker login -u cl00e9ment -p "$DOCKER_HUB_TOKEN"

build:
  stage: build
  script:
  - docker build --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host -t cl00e9ment/buildx .
  - docker run --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host cl00e9ment/buildx docker buildx inspect --bootstrap
  - docker push cl00e9ment/buildx

test:
  stage: test
  script:
  - docker run --add-host docker:`grep docker /etc/hosts | awk 'NR==1{print $1}'` --network host cl00e9ment/buildx docker buildx inspect --bootstrap

So what's happening?

At the end of the build, in the Dockerfile, I run docker buildx inspect --bootstrap to list the available platforms. It gives linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6. So it's all good.
After that, I run it again (just after the build and just before the push) and it still gives linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6.
However, on the test stage, when the image is freshly downloaded from Docker Hub on a clean environment, it gives linux/amd64, linux/386.

Why?

Solution

There is a lot of outdated and incorrect information on building multiarch images on GitLab CI unfortunately. The seems to change quite frequently as it's still an experimental feature. But as of the time of this post, this is how I got my multiarch build working on GitLab public runners (armv6, armv6, arm64, amd64):

First, one must build and push a Docker image containing the buildx binary. Here is the Dockerfile I am using for that:

FROM docker:latest
ARG BUILDX_VER=0.4.2
RUN mkdir -p /root/.docker/cli-plugins && \
    wget -qO ~/.docker/cli-plugins/docker-buildx \
    https://github.com/docker/buildx/releases/download/v${BUILDX_VER}/buildx-v${BUILDX_VER}.linux-amd64 && \
    chmod +x /root/.docker/cli-plugins/docker-buildx

The current GitLab runner image does not initialize the binfmt handlers correctly despite running the initialization code: https://gitlab.com/gitlab-org/gitlab-runner/-/blob/523854c8/.gitlab/ci/_common.gitlab-ci.yml#L91

So we have to do it in our pipeline. We refer to the comments in MR 1861 of the GitLab Runner code and add in the following magic sauce to our .gitlab-ci.yml:

before_script:
  - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Then we can run the rest of our pipeline script with docker login, docker buildx build --use, docker buildx build --push ... and so on.

Now the runner is ready to build for multiple architectures. My final .gitlab-ci.yml can be seen here: https://github.com/oofnikj/nuttssh/blob/multiarch/.gitlab-ci.yml