This output says that I'm running kubernetes with containerd as the container runtime:
k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-worker3 Ready <none> 12d v1.24.4+k3s1 10.16.24.123 <none> Ubuntu 20.04.2 LTS 5.15.0-48-generic containerd://1.6.6-k3s1
k8s-worker1 Ready <none> 12d v1.24.4+k3s1 10.16.24.121 <none> Ubuntu 20.04.2 LTS 5.13.0-44-generic containerd://1.6.6-k3s1
k8s-master Ready control-plane,master 12d v1.24.4+k3s1 10.16.24.120 <none> Ubuntu 20.04.4 LTS 5.15.0-46-generic containerd://1.6.6-k3s1
k8s-worker2 Ready <none> 12d v1.24.4+k3s1 10.16.24.122 <none> Ubuntu 20.04.2 LTS 5.13.0-44-generic containerd://1.6.6-k3s1
I'm deploying one of my pods, it gets scheduled on node k8s-worker3, and kubectl describe pods/mypod says the image was already on the node.
But when I run ctr on the node it shows that there NO images:
user@k8s-worker3:~$ sudo ctr images list
REF TYPE DIGEST SIZE PLATFORMS LABELS
And docker images doesn't show the correct version of the image.
Here's the processes running containerd:
user@k8s-worker3:~$ ps -ef | grep container
root 985 1 0 15:23 ? 00:00:00 /usr/bin/containerd
root 1106 1 0 15:23 ? 00:00:01 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 1312 1057 0 15:23 ? 00:00:12 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
root 1918 1 0 15:23 ? 00:00:03 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5abd3c3104abf812422757d9534c33475819cbf43b64845dd23f535193fed09e -address /run/k3s/containerd/containerd.sock
root 1970 1 0 15:23 ? 00:00:00 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id 8c339c15ab8abb3050810b5fcaed817363fcf3b06800b554711b1fa0e95006e2 -address /run/k3s/containerd/containerd.sock
root 3231 1 0 15:25 ? 00:00:01 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace k8s.io -id 354aa1df3c2a8492a4842efe3882785553501d77f61d9b5ef0bf3343ace2a518 -address /run/k3s/containerd/containerd.sock
user 4957 3094 0 15:48 pts/0 00:00:00 grep --color=auto container
So what is happening here? Where is the image on the node??
I found this command worked:
sudo crictl -r unix:///run/k3s/containerd/containerd.sock ps -a
So I needed to specify the Runtime Endpoint (option -r).