Rebuilding containers when needed

An application container has generally three parts:

• The base image that, in Linux case, contains the core POSIX tools and system libraries (e.g. FROM debian:stable)
• The runtime libraries or helper tools usually installed from some kind of package (e.g. RUN apt-get update && apt-get python3-numpy && …—for base python or java etc. there is usually a ready base container to use, but the application may still have additional dependencies).
  • The same issue applies to the dependencies linked into the application (e.g. external dependencies pulled in by maven or nuget etc.). It would be nice if there was a general solution that handled this too, but otherwise another tool-specific solution will be sought.
• The application itself (e.g. COPY …/application/ /usr/local/python3.8/dist-packages/application/ and set entrypoint).

The container that will go to the production environment is normally built on a build server. Now rebuilding it when changes are made to the application is obvious, and what CI servers are designed to do.

But the first two components may, and regularly do, have security advisories files against them, and when they do, the images should also be rebuilt and pushed to integration tests and then either the production environment should be updated too or the administrator notified to trigger it as soon as practical.

So is there some way or tool to help us set up suitable trigger to rebuild the container when there are updates to the base container or other dependencies? Docker default is Debian base images, but we would use another distribution if it had better support for this.

Note that just running docker build --pull --no-cache nightly is not a solution, because that will produce a new container even if the dependencies did not, in fact, change due to different timestamps of the installed files. And running just docker build --pull does not work either because either the cache is there, but then it only checks whether the Dockerfile has changed and does not actually check the package list, or the cache isn't there, because it is running on different build agent or the build agent was cleaned up as it generally should to get reliable builds.

It can use any container builder, not necessarily docker.

The answer is, as for many other docker-related issues, to ditch docker and switch to buildah. Both the buildah bud, which is drop-in replacement for docker build, and the buildah commit, which is for scripting the build by other means, have a --timestamp option that forces both the timestamp written to the manifest and the timestamps of files in the new layers to specified value. That seems to be the only nondeterminism from the tool itself; standard deterministic build techniques still need to be applied to the build of the application itself, but that's obviously out of buildah scope.