I have a running k8s cluster with two replicas of CoreDNS. But when i try enter the bash prompt of the POD it's throwing me below error
# kubectl exec -it coredns-5644d7b6d9-285bj -n kube-system sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "94f45da89fa5493a8283888464623788ef5e832dc31e0d89e427e71d86391fd6": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
But i am able to login to other pods without any issues. I tried with nsenter with kernel process ID it works but it only works for network related openrations like,
# nsenter -t 24931 -n ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 7a:70:99:aa:53:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.2/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::7870:99ff:feaa:536c/64 scope link
valid_lft forever preferred_lft forever
How to enter into this pod using kubectl and get rid of that error?
You can use the sidecar pattern following the instructions here: https://support.rancher.com/hc/en-us/articles/360041568712-How-to-troubleshoot-using-the-namespace-of-a-container#sidecar-container-0-2
In short, do this to find a node where a coredns pod is running:
kubectl -n kube-system get po -o wide | grep coredns
ssh to one of those nodes, then:
docker ps -a | grep coredns
Copy the Container ID to clipboard and run:
ID=<paste ID here>
docker run -it --net=container:$ID --pid=container:$ID --volumes-from=$ID alpine sh
You will now be inside the "sidecar" container and can poke around. I.e.
cat /etc/coredns/Corefile