Can I access a container in a pod in an AKS cluster as root?

I have a usecase in which I need root access to a container running on an aks node.

The kubectl exec command does not support a user option:

PS C:\Repos\GetShifting\work> kubectl exec -it kube-prometheus-stack-grafana-7d448f8457-wr9gg -c grafana -u root -- /bin/bash
error: unknown shorthand flag: 'u' in -u
See 'kubectl exec --help' for usage.

I can use kubectl debug as described here to access the host:

kubectl get pods kube-prometheus-stack-grafana-7d448f8457-wr9gg -o wide
kubectl debug node/aks-agentpool-14577650-vmss000033 -it --image=mcr.microsoft.com/cbl-mariner/busybox:2.0
chroot /host

Then on the host I want to do a docker exec command as a different user, like this:

docker ps
root@aks-agentpool-14577650-vmss0000033:/# sudo docker ps
sudo: docker: command not found
# so this one ain't going to work neither
docker exec -it -u root <dockerid> /bin/bash

Then I tried to show all containers running with the crictl command, and get the full container id from the correct container:

root@aks-agentpool-14577650-vmss000033:/# crictl ps | grep grafana
6439882b3402a       f9095e2f0444d       6 days ago          Running             grafana                           0                   75ba31cfe2863       kube-prometheus-stack-grafana-7d448f8457-wr9gg
658d77517b020       2e6ed1888609c       6 days ago          Running             grafana-sc-datasources            0                   75ba31cfe2863       kube-prometheus-stack-grafana-7d448f8457-wr9gg
fc35630b1f362       2e6ed1888609c       6 days ago          Running             grafana-sc-dashboard              0                   75ba31cfe2863       kube-prometheus-stack-grafana-7d448f8457-wr9gg
root@aks-agentpool-14577650-vmss000033:/# crictl ps --verbose --id 6439882b3402
ID: 6439882b3402a66761dde073c60721cca3ce89f9e2332a9ca4da3e8cbb268dec
PodID: 75ba31cfe28639a688f06693b7120faccef4bb1a724ed1e8cb01b1cb4bcdc881
Name: grafana
Attempt: 0
State: Running
Image: sha256:f9095e2f0444d50d16f3d1a1bb52ca4a6971ba7359fe78e0f0247c3dc70047cc
Created: 6 days ago
Labels:
        io.kubernetes.container.name -> grafana
        io.kubernetes.pod.name -> kube-prometheus-stack-grafana-7d448f8457-wr9gg
        io.kubernetes.pod.namespace -> vtxops
        io.kubernetes.pod.uid -> 5b01fe35-bbbb-46c1-a56d-cc156e426ec5
Annotations:
        io.kubernetes.container.hash -> 9661435a
        io.kubernetes.container.ports -> [{"name":"grafana","containerPort":3000,"protocol":"TCP"},{"name":"gossip-tcp","containerPort":9094,"protocol":"TCP"},{"name":"gossip-udp","containerPort":9094,"protocol":"UDP"}]
        io.kubernetes.container.restartCount -> 0
        io.kubernetes.container.terminationMessagePath -> /dev/termination-log
        io.kubernetes.container.terminationMessagePolicy -> File
        io.kubernetes.pod.terminationGracePeriod -> 30

And then I wanted to use runc to start a shell within the container:

root@aks-agentpool-14577650-vmss000003:/# runc --root /run/containerd/runc/k8s.io/ exec --tty --user 0 6439882b3402a66761dde073c60721cca3ce89f9e2332a9ca4da3e8cbb268dec sh
FATA[0000] nsexec-1[636535]: failed to open /proc/29605/ns/ipc: Permission denied
FATA[0000] nsexec-0[636532]: failed to sync with stage-1: next state: Success
ERRO[0000] exec failed: unable to start container process: error executing setns process: exit status 1

Then I found another commandline option, using ctr but I got another error there:

root@aks-agentpool-14577650-vmss000003:/# ctr -n k8s.io task exec --user 0 --exec-id 0 --fifo-dir /tmp -t 6439882b3402a66761dde073c60721cca3ce89f9e2332a9ca4da3e8cbb268dec sh
ctr: failed to unmount /tmp/containerd-mount3662884625: operation not permitted: failed to mount /tmp/containerd-mount3662884625: permission denied

I'm stuck here, is it possible to access a container in a pod in an AKS cluster as root?

PS: I know there are several questions like this out there but none of the solutions I found so far seem to work.

Solution

This is quite tricky - the solution is to run ctr with the parameter --user 0:0 instead of just --user 0. (I am not fully sure why the group id is needed here, but it works with it.)

So this should work:

ctr -n k8s.io task exec --user 0:0 --exec-id 0 --fifo-dir /tmp -t 6439882b3402a66761dde073c60721cca3ce89f9e2332a9ca4da3e8cbb268dec sh

Just tried this with my test cluster:

root@aks-usernodepl-14294476-vmss00000J:/# ctr -n k8s.io task exec -t --exec-id 0  87b20ec9ee77e1ff68efe4b9e7adbdba07fc40adc9ac7a65f301d140db9d03a0 id
uid=1002(cnb) gid=1000(cnb) groups=1000(cnb)
root@aks-usernodepl-14294476-vmss00000J:/# ctr -n k8s.io task exec -t --user 0:0 --exec-id 0  87b20ec9ee77e1ff68efe4b9e7adbdba07fc40adc9ac7a65f301d140db9d03a0 id
uid=0(root) gid=0(root) groups=0(root)