In my Kubernetes pods with mounted volumes, I don't seem to have access to the underlying disk device in the /dev folder. I need that for the XFS tools to work. I am running the cluster on DigitalOcean.
The example volume is mounted on /var/www. The output of df from a running pod is:
$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 158G 9.7G 142G 7% /
tmpfs 64M 0 64M 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc 1.0M 1.0M 0 100% /var/www
/dev/vda1 158G 9.7G 142G 7% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /sys/firmware
However, the output of lsblk does not reveal any such device /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc; it shows a /dev/sdb device instead.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1G 0 disk
sdb 8:16 0 1G 0 disk /var/www
sdc 8:32 0 1G 0 disk
sde 8:64 0 1G 0 disk
vda 254:0 0 160G 0 disk
|-vda1 254:1 0 160G 0 part /etc/apache2/sites-available
`-vda2 254:2 0 2M 0 part
vdb 254:16 0 472K 1 disk
None of the devices are avaiable in my pod:
$ ls -la /dev
total 4
drwxr-xr-x 5 root root 360 Oct 27 10:59 .
drwxr-xr-x 1 root root 4096 Oct 27 10:59 ..
lrwxrwxrwx 1 root root 11 Oct 27 10:59 core -> /proc/kcore
lrwxrwxrwx 1 root root 13 Oct 27 10:59 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Oct 27 10:59 full
drwxrwxrwt 2 root root 40 Oct 27 10:58 mqueue
crw-rw-rw- 1 root root 1, 3 Oct 27 10:59 null
lrwxrwxrwx 1 root root 8 Oct 27 10:59 ptmx -> pts/ptmx
drwxr-xr-x 2 root root 0 Oct 27 10:59 pts
crw-rw-rw- 1 root root 1, 8 Oct 27 10:59 random
drwxrwxrwt 2 root root 40 Oct 27 10:58 shm
lrwxrwxrwx 1 root root 15 Oct 27 10:59 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Oct 27 10:59 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Oct 27 10:59 stdout -> /proc/self/fd/1
-rw-rw-rw- 1 root root 0 Oct 27 10:59 termination-log
crw-rw-rw- 1 root root 5, 0 Oct 27 10:59 tty
crw-rw-rw- 1 root root 1, 9 Oct 27 10:59 urandom
crw-rw-rw- 1 root root 1, 5 Oct 27 10:59 zero
I also tried to change runAsUser, runAsGroup and added Linux capabilities such as SYS_ADMIN and SYS_RESOURCE - it didn't help.
When I set the container to run as privileged, I had access to the /dev/sdb and then I could symlink it to /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc so that the XFS tools can run.
However, it feels like a hack to create the symlink manually and also it only works with privileged containers:
SOURCE_LINK=$(df -h | grep -- "/var/www" | cut -d" " -f1)
TARGET_DEVICE=$(lsblk | grep -- "/var/www" | cut -d" " -f1)
mkdir -p $(dirname $SOURCE_LINK)
ln -s "/dev/$TARGET_DEVICE" "$SOURCE_LINK"
So my question is: is there some kind of configuration / settings / approach that would reveal the /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc in the pod, ideally in the non-privileged mode?
(This is related to another question of mine on how to enable the XFS quotas.)
This question was answered in another question of mine by Timo from DigitalOcean.
In order to get access to the device, you also need to mount the host's /dev directory to the container.
volumes:
- name: device-dir
hostPath:
path: /dev
and for the container:
volumeMounts:
- name: device-dir
mountPath: /dev
This removes the need to create the symlink manually as it already exists in the host. However, the device won't be writable. As explained in the Kubernetes documentation, the container needs to run as privileged and root in order to be able to write to such directories or the host's directories needs to have their permissions adjusted.