I have a small script running in a Pod that pokes for the latest App images (dashboard:development) in my registry and then pushes them to the Nodes running (via a daemonset).
This does work, as seen below.
Now, I would assume that once an App pod (like sp-pod-xx) requests this image, kubelet should not try to re-pull the image, even if imagePullPolicy: Always is set. As the docs say, kubelet compares the digest and only pulls, if there is a mismatch:
Always: every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet pulls the image with the resolved digest, and uses that image to launch the container.
But, even though the digests are identical (I did verify this), kubelet still re-pulls the image anyway. The App pod and the Daemonset pods are running on the same nodes too.
Any idea why?
Event logs:
4m5s Normal Killing pod/image-puller-ds-ldbfz
3m57s Normal SuccessfulCreate daemonset/image-puller-ds Created pod: image-puller-ds-fcmts
3m57s Normal SuccessfulCreate daemonset/image-puller-ds Created pod: image-puller-ds-fhhds
3m57s Normal Pulled pod/image-puller-ds-fhhds Successfully pulled image "dashboard:development" in 192.717161ms
3m57s Normal Pulling pod/image-puller-ds-fhhds Pulling image "dashboard:development"
3m56s Normal Started pod/image-puller-ds-fhhds Started container image-puller
3m56s Normal Created pod/image-puller-ds-fcmts Created container image-puller
3m56s Normal Created pod/image-puller-ds-fhhds Created container image-puller
3m56s Normal Started pod/image-puller-ds-fcmts Started container image-puller
3m56s Normal Pulled pod/image-puller-ds-fhhds Container image "pause:0.0.1" already present on machine
3m55s Normal Created pod/image-puller-ds-fcmts Created container pause
3m55s Normal SuccessfulDelete daemonset/image-puller-ds Deleted pod: image-puller-ds-xt9vv
3m55s Normal Pulled pod/image-puller-ds-fcmts Container image "pause:0.0.1" already present on machine
3m55s Normal Created pod/image-puller-ds-fhhds Created container pause
3m55s Normal Started pod/image-puller-ds-fhhds Started container pause
3m55s Normal Started pod/image-puller-ds-fcmts Started container pause
3m55s Normal Killing pod/image-puller-ds-xt9vv Stopping container pause
3m54s Normal Killing pod/image-puller-ds-wgwzh Stopping container pause
3m54s Normal SuccessfulDelete daemonset/image-puller-ds Deleted pod: image-puller-ds-wgwzh
3m25s Normal Pulling pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d Pulling image "dashboard:development"
3m25s Normal Pulled pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d Successfully pulled image "dashboard:development" in 220.610781ms
3m25s Normal Created pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d Created container sp-container-f3884032-1164-48e8-8213-c0c3856e573d
3m25s Normal Started pod/sp-pod-f3884032-1164-48e8-8213-c0c3856e573d Started container sp-container-f3884032-1164-48e8-8213-c0c3856e573d
Versions:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.12", GitCommit:"f941a31f4515c5ac03f5fc7ccf9a330e3510b80d", GitTreeState:"clean", BuildDate:"2022-11-09T17:12:33Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
The docs are wrong. You can read the source code here. The policy PullIfNotPresent would check if the image exists, but Always skips directly to pulling the image.