[SOLVED] Multi-Platform Docker Images Interfering With Each Other During Build

Multi-Platform Docker Images Interfering With Each Other During Build

I'm trying to build Postgres Docker images for both amd64 & arm64 with a pre-populated database of AU addresses; using an M2 MacBook with Docker Engine 27.4.0.

When the build gets to the stage of running pg_restore to load the data in both images; I get duplicated ID failures creating primary keys. These IDs are not duplicated in the dump files (they're used by many users and I've loaded them myself without issue - FYI see https://github.com/minus34/gnaf-loader for info on the data itself)

I can only conclude the databases in the 2 images are interfering with each other, even though they should be running on their own internal port 5432.

EDIT: Building each platform separately with the code below works fine.

Build code:

!/usr/bin/env bash 
            
# 1. launch buildx
docker buildx create --name gnafloader_test_builder --use
docker buildx inspect --bootstrap
            
# 2. build images
docker buildx build --no-cache --platform linux/arm64,linux/amd64 --tag minus34/gnafloader_test:latest --tag minus34/gnafloader_test:202411 -f ${DOCKER_FOLDER}/Dockerfile . --load

Dockerfile excerpt:

FROM debian:bookworm-slim

# install Postgres with PostGIS

...

# copy and restore GNAF & Admin Boundary Postgres dump files
RUN mkdir -p /data
WORKDIR /data

ADD gnaf-202411.dmp .
ADD admin-bdys-202411.dmp .

RUN /etc/init.d/postgresql start \
    && pg_restore -Fc -d postgres -h 127.0.0.1 -p 5432 -U postgres /data/gnaf-202411.dmp \
    && /etc/init.d/postgresql stop \
    && rm /data/gnaf-202411.dmp

RUN /etc/init.d/postgresql start \
    && pg_restore -Fc -d postgres -h 127.0.0.1 -p 5432 -U postgres /data/admin-bdys-202411.dmp \
    && /etc/init.d/postgresql stop \
    && rm /data/admin-bdys-202411.dmp   

# enable external access to postgres - WARNING: these are insecure settings! Edit these to restrict access
RUN echo "host all  all    0.0.0.0/0  md5" >> /etc/postgresql/16/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/16/main/postgresql.conf

EXPOSE 5432

# set user for postgres startup
USER postgres

# Start postgres when starting the container
CMD ["/usr/lib/postgresql/16/bin/postgres", "-D", "/var/lib/postgresql/16/main", "-c", "config_file=/etc/postgresql/16/main/postgresql.conf"]

Is there any way to avoid this using Docker BuildX or should I try Podman?

Solution

After more research on the issue - I solved the problem using a Podman manifest. It processes the different platforms sequentially by default. Avoiding the parallel processing issue above.

podman manifest create localhost/gnafloader
podman build --quiet --squash --platform linux/amd64,linux/arm64/v8 --manifest localhost/gnafloader .
podman manifest push --compression-level 9 localhost/gnafloader docker://docker.io/minus34/gnafloader:latest

The full solution is here:

https://github.com/minus34/gnaf-loader/blob/master/supporting-files/quarterly_processing/04-create-podman-images.sh

https://github.com/minus34/gnaf-loader/blob/master/docker/Dockerfile