OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown

I have read many links similar to my issue, but none of them were helping me to resolve the issue.

Similar Links:

1. Failed to exec into the container due to permission issue after executing 'systemctl daemon-reload'
2. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
3. CI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
4. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
5. Fail to execute docker exec
6. OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

Problem Description:

I have created a new Kubernetes cluster using Kubespray. When I wanted to execute some commands in one of containers I faced to the following error:

Executed Command

kubectl exec -it -n rook-ceph rook-ceph-tools-68d847b88d-7kw2v -- sh

Error:

OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/1: operation not permitted: unknown command terminated with exit code 126

I have also logged in to the node, which runs the pod, and try executing the container using docker exec command, but the error was not changed.

Workarounds:

• As I have found, the error code (126) implies that the permissions are insufficient, but I haven't faced this kind of error (like executing sh) in Docker or Kubernetes.

• I have also checked whether SELinux is enabled or not (as it has been said in the 3rd link).

  apt install policycoreutils
  sestatus
  # Output
  SELinux status:                 disabled
  

• In the 5th link, it was said to check whether you have updated the kernel, and I didn't upgrade anything on the nodes.

  id; stat /dev/pts/0
  # output
  uid=0(root) gid=0(root) groups=0(root)
    File: /dev/pts/0
    Size: 0               Blocks: 0          IO Block: 1024   character special file
  Device: 18h/24d Inode: 3           Links: 1     Device type: 88,0
  Access: (0600/crw-------)  Uid: (    0/    root)   Gid: (    5/     tty)
  Access: 2022-08-21 12:01:25.409456443 +0000
  Modify: 2022-08-21 12:01:25.409456443 +0000
  Change: 2022-08-21 11:54:47.474457646 +0000
   Birth: -
  

• Also tried /bin/sh instead of sh or /bin/bash, but not worked and the same error occurred.

Can anyone help me to find the root cause of this problem and then solve it?

This issue may relate to docker, first drain your node.

kubectl drain <node-name>

Second, SSH to the node and restart docker service.

systemctl restart docker.service

At the end try to execute your command.