Problems with RabbitMQ in Nomad with docker volumes

I am attempting to get RabbitMQ to run in Nomad using Docker. However I have stumbled into some problems related to permissions. When attempting to run the Job in Nomad I either get this error:

sed: preserving permissions for ‘/etc/rabbitmq/sedpR1m3w’: Operation not permitted
sed: preserving permissions for ‘/etc/rabbitmq/sedEc0Idz’: Operation not permitted
/usr/local/bin/docker-entrypoint.sh: line 250: /etc/rabbitmq/rabbitmq.conf: Permission denied
touch: cannot touch '/etc/rabbitmq/rabbitmq.conf': Permission denied

WARNING: '/etc/rabbitmq/rabbitmq.conf' is not writable, but environment variables have been provided which request that we write to it
  We have copied it to '/tmp/rabbitmq.conf' so it can be amended to work around the problem, but it is recommended that the read-only source file should be modified and the environment variables removed instead.

/usr/local/bin/docker-entrypoint.sh: line 250: /tmp/rabbitmq.conf: Permission denied

or this error:

chmod: changing permissions of '/var/lib/rabbitmq/.erlang.cookie': Operation not permitted

I have setup volumes so that RabbitMQ data can be preserved. These volumes is pointing to an SMB share on a Windows Server box elsewhere on the network. I have added the following to /etc/ftstab for auto mounting:

//DC02/Nomad /mnt/winshare cifs credentials=/home/linuxnomad/.smbcreds,uid=995,gid=993,file_mode=0777,dir_mode=0777 0 0

This is what the Job spec looks like:

job "rabbitmq03" {
  datacenters = ["techtest"]
  type        = "service"
  
  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

    constraint {
    attribute = "${attr.unique.hostname}"
    value     = "nomadlinux03"
  }
  
  group "rabbitmq" {
    network {
        mode = "cni/prod"
      hostname = "RabbitMqNOMAD03"
    }
    
    service {
      name         = "${JOB}"
      port         = 5672
      address_mode = "alloc"
      check {
        type         = "http"
            port         = 15672
        path         = "/api/health/checks/local-alarms"
        interval     = "3s"
        timeout      = "2s"
        address_mode = "alloc"
        header {
          Authorization = ["Basic Z3Vlc3Q6Z3Vlc3Q="]
        }
      }
    }

    task "rabbitmq" {
      driver = "docker"
      
      config {
        privileged     = false
        image          = "rabbitmq:3.8.12-management"
        auth_soft_fail = true
        
        volumes = [
          "/mnt/winshare/RabbitMQ03/data:/var/lib/rabbitmq/mnesia",
          "/mnt/winshare/RabbitMQ03/config:/etc/rabbitmq",
          "/mnt/winshare/RabbitMQ03/log:/var/log/rabbitmq"
        ]
      }

      env {
        HOSTNAME = "RabbitMqNOMAD"
        RABBITMQ_DEFAULT_USER = "guest"
        RABBITMQ_DEFAULT_PASS = "guest"
        RABBITMQ_ERLANG_COOKIE = "testsecret"    
      }

      resources {
        cpu    = 1001
        memory = 6144
      }
    }
  }
}

I did make sure to mount the SMB share with the Nomad user rights, so my expectation would be that it's fine, but perhaps I'm missing something?

Solution

I did make sure to mount the SMB share with the Nomad user rights

You are running a docker container. Nomad user rights are irrelevant, as long as it can access docker daemon.

perhaps I'm missing something?

Samba and cifs has it's own permissions, and you are forcing uid=995,gid=993,file_mode=0777,dir_mode=0777.

Research docker containers and what is user virtualization. Your error is unrelated to Nomad. Research samba permisions and the specific docker container and application you are running, i.e. the rabbitmq:3.8.12-management docker continaer, for what permissions it expects. Additionally, research the standard linux file permission model.

(Also, I think, bind-mounting a subdirectory of CIFS mount might not work as expected, but this is a guess.)

The container changes to rabbitmq user on entrypoint https://github.com/docker-library/rabbitmq/blob/master/docker-entrypoint.sh#L10 .