I was reading about the central concepts in Docker. The following line from Docker's own guides confused me.¹

Images are immutable. Once an image is created, it can't be modified. You can only make a new image or add changes on top of it.

I initially doubted my interpretation of this informal explanation. But apparently, any change in an image does indeed invalidate all subsequent layers.² There are even checksums that cause upper layers to reject any underlying change.^{3 4}

In my (ignorant) imagination, it is possible in certain cases to modify an underlying layer and retain an already built upper layer, without resulting in any deviant behaviour relative to that under the scenario of the upper layer being rebuilt. Let's say hypothetically that a new line is added to a Python file residing in the lower layer, in a way that does not affect its semantics. Is it necessary in this case to regard the subsequent layers as invalid and rebuild them?

I'm thinking from the assumption that this is not a limitation of OverlayFS,^a but an arbitrary restriction imposed by Docker to prevent negligence. There are some online discussions outlining sketchy ways to modify images without rebuilding. Of course, there is no reason to doubt that switching a Ubuntu base image for a Fedora one would wreck havoc.

Notes and references

^a Documentation written by Neil Brown in kernel.org says, "Offline changes, when the overlay is not mounted, are allowed to either the upper or the lower trees."⁵

1. https://docs.docker.com/get-started/docker-concepts/the-basics/what-is-an-image/
2. https://docs.docker.com/get-started/docker-concepts/building-images/using-the-build-cache/
3. https://docs.docker.com/reference/cli/docker/image/pull/
4. https://stackoverflow.com/a/43415658/24232001
5. https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt

Is it necessary in this case to regard the subsequent layers as invalid and rebuild them?

Docker build tooling has no way to know which changes will not affect later build steps. So once a step changes, future steps will not use the existing build cache. And when a step is run without the cache, things like timestamps result in the packaging of the layer having a different hash.

Could a layer be directly modified? Yes, I've done this with my own tooling. The "image is immutable" rule still applies. Changing the image changes layers, those layer digests change the resulting image digest, and a container should never be able to modify the underlying layer filesystem (this would be a security vulnerability).

There is an exception to the rule, COPY --link was added to buildkit to copy content without considering the existing state of the image. This allows later layers to reuse the cache from an unchanged stage in a multi-stage build.