dockerdocker-networkdocker-swarm-modedocker-stack

Docker - Inter stack network issues in a single node swarm


I have a single node swarm on Ubuntu 20. I have setup 3 stacks, all using default overlay network. Per documentation containers should be able to communicate with each other without any special setup.

But in my case containers across stacks are not able to resolve each other. Different containers inside same stack work just fine. Looking at the IP segment used, they should be able to talk to each other. Any idea on how to troubleshoot why they are not communicating?

Basic setup

uname -a
Linux ubuntu20 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux


docker stack ls
NAME                SERVICES            ORCHESTRATOR
management          1                   Swarm
tech                3                   Swarm
web                 1                   Swarm

docker service ls
ID                  NAME                      MODE                REPLICAS            IMAGE                           PORTS
cla322jwbgm3        management_portainer-ce   replicated          1/1                 portainer/portainer-ce:latest   *:8000->8000/tcp, *:9000->9000/tcp
qlts0otlc4c8        tech_grafana              replicated          1/1                 grafana/grafana:latest          *:3000->3000/tcp
yujqvwim2okb        tech_influxdb             replicated          1/1                 influxdb:latest                 *:8086->8086/tcp
v14haj0oa8dc        tech_mosquitto            replicated          1/1                 eclipse-mosquitto:latest        *:1883->1883/tcp
3dra8oyo7swo        web_nginx                 replicated          1/1                 nginx:latest                    *:80->80/tcp


docker container ls
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                NAMES
9025b777f04c        portainer/portainer-ce:latest   "/portainer"             29 minutes ago      Up 29 minutes       8000/tcp, 9000/tcp   management_portainer-ce.1.w910sug8vaormu1pgs62wrj2j
918705a22e73        influxdb:latest                 "/entrypoint.sh infl…"   29 minutes ago      Up 29 minutes       8086/tcp             tech_influxdb.1.4d47iflbvoybnzsl1lquf89ej
4911abc53cf0        grafana/grafana:latest          "/run.sh"                29 minutes ago      Up 29 minutes       3000/tcp             tech_grafana.1.oyoot8zkxl92yi05s1ac7nano
a00639d4aa76        eclipse-mosquitto:latest        "/docker-entrypoint.…"   29 minutes ago      Up 29 minutes       1883/tcp             tech_mosquitto.1.toob1isapbmy0h0i4wa97uqq6
ca60cb673379        nginx:latest                    "/docker-entrypoint.…"   30 minutes ago      Up 30 minutes       80/tcp               web_nginx.1.mgyk64ggouqi9pyzgkyknmuk4

Network

docker network create   --driver overlay   --ingress   --subnet=10.11.0.0/16   --gateway=10.11.0.1 ingress


docker inspect network ingress

[
    {
        "Name": "ingress",
        "Id": "9ykfwv0ahhk7a9s8g8a2nh53q",
        "Created": "2021-02-07T13:52:09.317600628-05:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.11.0.0/16",
                    "Gateway": "10.11.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "4911abc53cf0021e0fbad4392a1666e17d500167589460a77bdbd73bc52a75a1": {
                "Name": "tech_grafana.1.oyoot8zkxl92yi05s1ac7nano",
                "EndpointID": "c9d609ffe49a7aad7be596f6f36bf9b494af109a4c530fc8b5393d0ff3c16753",
                "MacAddress": "02:42:0a:0b:00:08",
                "IPv4Address": "10.11.0.8/16",
                "IPv6Address": ""
            },
            "9025b777f04c86742b87308374cda51b11530556040faad623dd5db51f246de4": {
                "Name": "management_portainer-ce.1.w910sug8vaormu1pgs62wrj2j",
                "EndpointID": "bfd0613488921f39d69f4dc4e45e1f2b1b80b7c41a411040f1c2f7518c06f3c6",
                "MacAddress": "02:42:0a:0b:00:0c",
                "IPv4Address": "10.11.0.12/16",
                "IPv6Address": ""
            },
            "918705a22e73c0f6ea81b063bd95db964d74cbb0f99b5fe19bbfb0cb352b0bfc": {
                "Name": "tech_influxdb.1.4d47iflbvoybnzsl1lquf89ej",
                "EndpointID": "07a5cac92d0c5fc53e93bf6c049f7b8977dd31c1a2d6fc27a033f1769370c53b",
                "MacAddress": "02:42:0a:0b:00:0a",
                "IPv4Address": "10.11.0.10/16",
                "IPv6Address": ""
            },
            "a00639d4aa76485b305b41e4052fe955eef4ee2c0f0e90722097d6a7995ca9bc": {
                "Name": "tech_mosquitto.1.toob1isapbmy0h0i4wa97uqq6",
                "EndpointID": "080d882f2cc09a011261a11c6f46d8c04f26a2329d00f5fa530133bf67a4d8c0",
                "MacAddress": "02:42:0a:0b:00:06",
                "IPv4Address": "10.11.0.6/16",
                "IPv6Address": ""
            },
            "ca60cb6733797cd6867fb7146217f92ad18d1f759c86dbb65f0804199129651a": {
                "Name": "web_nginx.1.mgyk64ggouqi9pyzgkyknmuk4",
                "EndpointID": "019506a9da152f923600580121c373e7a74ef99cd05bf3c5094ab92dfe39a472",
                "MacAddress": "02:42:0a:0b:00:03",
                "IPv4Address": "10.11.0.3/16",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "b89f39cca8cfc6553385247315adaf543900a4e56ca769c2989e6ad63baa0ebc",
                "MacAddress": "02:42:0a:0b:00:04",
                "IPv4Address": "10.11.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4104"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "933a5c2c1e76",
                "IP": "192.168.1.103"
            }
        ]
    }
]

Inter container in same stack works, but across stack fails.

#from tech stack->web stack fails, inside works
docker exec -it 918705a22e73 bash
root@918705a22e73:/# ping nginx
ping: nginx: Name or service not known

root@918705a22e73:/# ping grafana
PING grafana (10.0.9.5) 56(84) bytes of data.
64 bytes from 10.0.9.5 (10.0.9.5): icmp_seq=1 ttl=64 time=0.099 ms
^C
--- grafana ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.099/0.099/0.099/0.000 ms

#from web->web stack works, web->tech fails
root@ca60cb673379:/# curl grafana
curl: (6) Could not resolve host: grafana
root@ca60cb673379:/# curl nginx
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
root@ca60cb673379:/# curl influxdb
curl: (6) Could not resolve host: influxdb
root@ca60cb673379:/#

Solution

  • Soon I figured (the hard way) default overlay network connected does not offer interstack communication. For that to work you will need to create a new overlay network and associate the services to it.

    #create shared network for stacks
    docker network create --driver=overlay --attachable shared_overlay
    
        networks:
        - default
        - shared_overlay