Given Docker version 24.0.7-ce, build 311b9ff0aa93, and the following Dockerfile:
# syntax=docker/dockerfile:1.7-labs
FROM ubuntu AS base
# do some file changes here
RUN touch /opt/foo
FROM scratch AS final
COPY --from=base --exclude=/opt / /
COPY --from=base /opt /opt
$ docker buildx build -t foo .
...
=> ERROR [final 1/2] COPY --from=base --exclude=/opt / / 0.0s
------
> [final 1/2] COPY --from=base --exclude=/opt / /:
------
Dockerfile:8
--------------------
6 |
7 | FROM scratch AS final
8 | >>> COPY --from=base --exclude=/opt / /
9 | COPY --from=base /opt /opt
10 |
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref a6b7e3f0-4303-4882-9783-1e7a9ca68479::00cybyw1mr8d6grtvn00sat40: "/usr/share/doc/libapt-pkg6.0t64/NEWS.Debian.gz": not found
My best guess for the issue (given some similar errors when trying other things) is that docker is erroneously trying to resolve symlinks instead of copying/using the symlink itself, and it breaks.
In particular, in the above image the named file does not exist but there is a symlink /usr/share/doc/apt/NEWS.Debian.gz that points to it.
I think this is obviously a bug in buildkit, but is there a workaround for it? The goal is to take an existing image and "flatten" most of its layers, but keeping certain directories on their own layer, to avoid huge layer sizes. (This is obviously silly in this particular example; the real case is more complicated.)
I don't want to delete or "fix" the broken symlinks; I want to preserve them as-is. Given that they came from the upstream image in the first place they're presumably supposed to be tolerated.
Side note: during the course of testing this I also discovered that doing COPY --from=base /bin /opt / results in copying the contents of the directories and not the directories themselves (i.e. things end up flattened into the root directory and not under bin), which seems unexpected and contrary to how cp works. Is there a workaround for this too?
Edit: it looks like using COPY --from=base --parents /bin /opt / works more as expected -- provided it doesn't contain any broken symlinks. This doesn't seem like a good workaround for other cases where the complete prefix isn't intended, though.
It looks like this is this bug and there isn't really any good workaround except naming individual directories, which will create more layers than intended.
So, replacing:
COPY --from=base --exclude=/opt / /
with:
COPY --from=base /bin /bin
COPY --from=base /boot /boot
COPY --from=base /dev /dev
COPY --from=base /etc /etc
COPY --from=base /home /home
COPY --from=base /lib /lib
COPY --from=base /lib64 /lib64
COPY --from=base /media /media
COPY --from=base /mnt /mnt
COPY --from=base /proc /proc
COPY --from=base /root /root
COPY --from=base /run /run
COPY --from=base /sbin /sbin
COPY --from=base /srv /srv
COPY --from=base /sys /sys
COPY --from=base /tmp /tmp
COPY --from=base /usr /usr
COPY --from=base /var /var
(And then finishing up with copying /opt separately too, as before.)
This works, but is unsatisfying -- and fragile, if something adds additional root-level directories or files.
Some of these could be merged using --parents, but as noted in the original question this still breaks if there are any broken symlinks in those directories, so is still fragile.