TL;DR: I have a Dockerfile which I want to use to build some Rust code (with native dependencies) into as small as possible of a multi-platform image (for both linux/amd64 and linux/arm64) via the following command:
docker build --platform linux/amd64,linux/arm64 .
When I run this on my M1 MacBook, the linux/arm64 part succeeds but the linux/amd64 part fails, and when I run it on an x86 Linux machine in GitHub Actions, the linux/amd64 part succeeds but the linux/arm64 part fails. I've enabled containerd (and QEMU in CI) so I can build other multi-platform images just fine; this seems specific to building Rust code with native dependencies targeting musl.
I have a Rust binary crate that depends on Wasmtime; here's my Cargo.toml. (Note that if I remove wasmtime from my [dependencies], this error no longer occurs.)
[package]
name = "foo"
edition = "2021"
[dependencies]
wasmtime = "29"
I have this Dockerfile to build it; I'm using musl to build a static binary so I can pass it to the second stage which is built FROM scratch:
FROM rust
ARG TARGETARCH
RUN apt-get update && apt-get install -y musl-tools
WORKDIR /root
COPY . .
RUN rustup target add "$(./target.py $TARGETARCH)"
RUN cargo build --target "$(./target.py $TARGETARCH)"
RUN cp "target/$(./target.py $TARGETARCH)/debug/foo" .
FROM scratch
COPY --from=0 /root/foo /root/foo
ENTRYPOINT ["/root/foo"]
The target.py script just translates a Docker TARGETARCH to a Rust target:
#!/usr/bin/env python3
import sys
targets = {
"amd64": "x86_64-unknown-linux-musl",
"arm64": "aarch64-unknown-linux-musl",
}
print(targets[sys.argv[1]])
When I'm just building natively, everything works fine:
docker build . -t foo && docker run --rm foo
But if I attempt to build a multi-platform image, it fails:
docker build --platform linux/amd64 .
For instance, here is the most relevant portion of the error output when building on my M1 MacBook:
error occurred in cc-rs: Command ZERO_AR_DATE="1" "musl-ar" "cq" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/libzstd.a" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/44ff4c55aa9e5133-debug.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/44ff4c55aa9e5133-entropy_common.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/44ff4c55aa9e5133-error_private.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/44ff4c55aa9e5133-fse_decompress.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/44ff4c55aa9e5133-pool.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/44ff4c55aa9e5133-threading.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/44ff4c55aa9e5133-zstd_common.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-fse_compress.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-hist.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-huf_compress.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_compress.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_compress_literals.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_compress_sequences.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_compress_superblock.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_double_fast.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_fast.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_lazy.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_ldm.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstd_opt.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/fb80479a5fb81f6a-zstdmt_compress.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/88f362f13b0528ed-huf_decompress.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/88f362f13b0528ed-zstd_ddict.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/88f362f13b0528ed-zstd_decompress.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/88f362f13b0528ed-zstd_decompress_block.o" "/root/target/x86_64-unknown-linux-musl/debug/build/zstd-sys-7442e8fce933a8d2/out/7faed3f8272f2313-huf_decompress_amd64.o" with args musl-ar did not execute successfully (status code exit status: 127).
I've also created a GitHub repository containing the full set of files to reproduce this example.
Is it possible to modify my Dockerfile to build this multi-platform image on one machine via the single docker build command I listed at the top (which already works for all my other Dockerfiles)? I know that there are other ways to build cross-platform musl binaries (cross works in this case) but I want to keep my Docker build infrastructure consistent with the rest of my larger project.
TL;DR: add this to your Dockerfile somewhere before the cargo build command:
ENV TARGET_AR=ar
In this particular case, here is the chain of dependencies to the issue:
In its build.rs script, the zstd-sys crate makes this chain of calls:
That last method returns basically the same thing as cc::Build::get_archiver, which is usually just ar. But not always:
x86_64-unknown-linux-musl in a linux/amd64 Docker image on an M1 MacBook, it's musl-ar instead, which does not exist.aarch64-unknown-linux-musl in a linux/arm64 Docker image on an x86 Linux machine, it's aarch64-linux-musl-ar instead, which also does not exist.The TARGET_AR environment variable can be used to override cc's default archiver choice. It's not documented anywhere, but is implemented here as part of a generic "{kind}_{var_base}" pattern where in this case kind is "TARGET" and var_base is "AR".
I've opened issue rust-lang/cc-rs#1399 to see whether the cc maintainers consider this a bug.