Running LINSTOR in Docker Swarm

I am currently trying out linstor in my lab. I am trying to setup a separation of compute and storage node. Storage node that runs linstor whereas Compute node is running Docker Swarm or K8s. I have setup 1 linstor node and 1 docker swarm node in this testing. Linstor node is configured successfully.

Linstor Node

DRBD 9.1.2

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node       ┊ Driver   ┊ PoolName       ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ instance-2 ┊ DISKLESS ┊                ┊              ┊               ┊ False        ┊ Ok    ┊            ┊
┊ pd-std-pool          ┊ instance-2 ┊ LVM_THIN ┊ vg/lvmthinpool ┊   199.80 GiB ┊    199.80 GiB ┊ True         ┊ Ok    ┊            ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
# linstor node list
╭─────────────────────────────────────────────────────────────╮
┊ Node       ┊ NodeType ┊ Addresses                ┊ State    ┊
╞═════════════════════════════════════════════════════════════╡
┊ instance-2 ┊ COMBINED ┊ 10.100.0.29:3366 (PLAIN) ┊ Online   ┊
╰─────────────────────────────────────────────────────────────╯

Docker Node

In another node, I have Docker Swarm running. This node does not have any tools installed such as drbd, drbdtop, drbdsetup ...etc. Technically is running a minimal installation that is sufficient to run only Docker to keep it lightweight. Docker version is 20.10.3. I have also installed the linstor docker volume written in golang.

Below is my /etc/linstor/docker-volume.conf and docker volume plugin installed in my Docker Swarm node

$ docker plugin ls
ID             NAME                                  DESCRIPTION                        ENABLED
6300029b3178   linbit/linstor-docker-volume:latest   Linstor volume plugin for Docker   true

$ cat /etc/linstor/docker-volume.conf
[global]
controllers = linstor://instance-2
fs = xfs

I got an error when trying to use the volume created by linstor. I have confirmed I can ping linstor controller at instance-2 and have all ports open in the firewall. Here is the error and the step to reproduce

$ docker volume create -d linbit/linstor-docker-volume:latest  --name=first --opt size=20 --opt replicas=1 --opt storage-pool=pd-std-pool
$ docker volume ls
DRIVER                                VOLUME NAME
local                                 64f864db31990baa6b790dde34513a7f6fc466ca0c5e72ffab7024365a9f45da
linbit/linstor-docker-volume:latest   first
$ docker volume inspect first
[
    {
        "CreatedAt": "0001-01-01T00:00:00Z",
        "Driver": "linbit/linstor-docker-volume:latest",
        "Labels": {},
        "Mountpoint": "",
        "Name": "first",
        "Options": {
            "replicas": "1",
            "size": "20",
            "storage-pool": "pd-std-pool"
        },
        "Scope": "global"
    }
]
$ docker run --rm -it -v first:/data alpine sh
docker: Error response from daemon: error while mounting volume '': VolumeDriver.Mount: 404 Not Found.
ERRO[0000] error waiting for container: context canceled

Questions

Do I need to install drbd-utils in my Docker Swarm for it to work?
What is the error VolumeDriver.Mount 404 Not Found means ?

Solution

LINSTOR manages storage in a cluster of nodes replicating disk space inside a LVM or ZFS volume (or bare partition I'd say) by using DRDB (Distributed Replicated Block Device) to replicate data across the nodes, as per the official docs:

"LINSTOR is a configuration management system for storage on Linux systems. It manages LVM logical volumes and/or ZFS ZVOLs on a cluster of nodes. It leverages DRBD for replication between different nodes and to provide block storage devices to users and applications. It manages snapshots, encryption and caching of HDD backed data in SSDs via bcache."

So I'd say yes, you really need to have the driver on every node on which you want to use the driver (I did see Docker's storage plugin try to mount the DRBD volume locally)

However, you do not necessarily need to have the storage space itself on the compute node, since you can mount a diskless DRBD resource from volumes that are replicated on separate nodes so I'd say your idea should work, unless there is some bug in the driver itself I didn't discover yet: your compute node(s) needs to be registered as being a diskless node for all the required pools (I didn't try this but remember reading it's not only possible but recommended for some types of data migrations).

Of course if you don't have more than 1 storage nodes you don't gain much from using LINSTOR/drbd (node or disk failure will leave you diskless). My use case for it was to have replicated storage across different servers in different datacenters, so that the next time one burns to a crisp 😅 I can have my data and containers running after minutes instead of several days...