dockerdocker-composedocker-swarmdocker-swarm-mode

How to directly mount NFS share/volume in container using docker compose v3


I have a compose file with v3 where there are 3 services sharing/using the same volume. While using swarm mode we need to create extra containers & volumes to manage our services across the cluster.

I am planning to use NFS server so that single NFS share will get mounted directly on all the hosts within the cluster.

I have found below two ways of doing it but it needs extra steps to be performed on the docker host -

Is there a standard way where i can directly use/mount NFS share using docker compose v3 by performing only few/no steps(I understand that "nfs-common" package is required anyhow) on the docker host?


Solution

  • After discovering that this is massively undocumented,here's the correct way to mount a NFS volume using stack and docker compose.

    The most important thing is that you need to be using version: "3.2" or higher. You will have strange and un-obvious errors if you don't.

    The second issue is that volumes are not automatically updated when their definition changes. This can lead you down a rabbit hole of thinking that your changes aren't correct, when they just haven't been applied. Make sure you docker rm VOLUMENAME everywhere it could possibly be, as if the volume exists, it won't be validated.

    The third issue is more of a NFS issue - The NFS folder will not be created on the server if it doesn't exist. This is just the way NFS works. You need to make sure it exists before you do anything.

    (Don't remove 'soft' and 'nolock' unless you're sure you know what you're doing - this stops docker from freezing if your NFS server goes away)

    Here's a complete example:

    [root@docker docker-mirror]# cat nfs-compose.yml
    version: "3.2"
    
    services:
      rsyslog:
        image: jumanjiman/rsyslog
        ports:
          - "514:514"
          - "514:514/udp"
        volumes:
          - type: volume
            source: example
            target: /nfs
            volume:
              nocopy: true
    volumes:
      example:
        driver_opts:
          type: "nfs"
          o: "addr=10.40.0.199,nolock,soft,rw"
          device: ":/docker/example"
    
    
    
    [root@docker docker-mirror]# docker stack deploy --with-registry-auth -c nfs-compose.yml rsyslog
    Creating network rsyslog_default
    Creating service rsyslog_rsyslog
    [root@docker docker-mirror]# docker stack ps rsyslog
    ID                  NAME                IMAGE                       NODE                DESIRED STATE       CURRENT STATE                     ERROR               PORTS
    tb1dod43fe4c        rsyslog_rsyslog.1   jumanjiman/rsyslog:latest   swarm-4             Running             Starting less than a second ago
    [root@docker docker-mirror]#
    

    Now, on swarm-4:

    root@swarm-4:~# docker ps
    CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS               NAMES
    d883e0f14d3f        jumanjiman/rsyslog:latest   "rsyslogd -n -f /e..."   6 seconds ago       Up 5 seconds        514/tcp, 514/udp    rsyslog_rsyslog.1.tb1dod43fe4cy3j5vzsy7pgv5
    root@swarm-4:~# docker exec -it d883e0f14d3f df -h /nfs
    Filesystem                Size      Used Available Use% Mounted on
    :/docker/example          7.2T      5.5T      1.7T  77% /nfs
    root@swarm-4:~#
    

    This volume will be created (but not destroyed) on any swarm node that the stack is running on.

    root@swarm-4:~# docker volume inspect rsyslog_example
    [
        {
            "CreatedAt": "2017-09-29T13:53:59+10:00",
            "Driver": "local",
            "Labels": {
                "com.docker.stack.namespace": "rsyslog"
            },
            "Mountpoint": "/var/lib/docker/volumes/rsyslog_example/_data",
            "Name": "rsyslog_example",
            "Options": {
                "device": ":/docker/example",
                "o": "addr=10.40.0.199,nolock,soft,rw",
                "type": "nfs"
            },
            "Scope": "local"
        }
    ]
    root@swarm-4:~#