I have a single node swarm on Ubuntu 20. I have setup 3 stacks, all using default overlay network. Per documentation containers should be able to communicate with each other without any special setup.
But in my case containers across stacks are not able to resolve each other. Different containers inside same stack work just fine. Looking at the IP segment used, they should be able to talk to each other. Any idea on how to troubleshoot why they are not communicating?
Basic setup
uname -a
Linux ubuntu20 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
docker stack ls
NAME SERVICES ORCHESTRATOR
management 1 Swarm
tech 3 Swarm
web 1 Swarm
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
cla322jwbgm3 management_portainer-ce replicated 1/1 portainer/portainer-ce:latest *:8000->8000/tcp, *:9000->9000/tcp
qlts0otlc4c8 tech_grafana replicated 1/1 grafana/grafana:latest *:3000->3000/tcp
yujqvwim2okb tech_influxdb replicated 1/1 influxdb:latest *:8086->8086/tcp
v14haj0oa8dc tech_mosquitto replicated 1/1 eclipse-mosquitto:latest *:1883->1883/tcp
3dra8oyo7swo web_nginx replicated 1/1 nginx:latest *:80->80/tcp
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9025b777f04c portainer/portainer-ce:latest "/portainer" 29 minutes ago Up 29 minutes 8000/tcp, 9000/tcp management_portainer-ce.1.w910sug8vaormu1pgs62wrj2j
918705a22e73 influxdb:latest "/entrypoint.sh infl…" 29 minutes ago Up 29 minutes 8086/tcp tech_influxdb.1.4d47iflbvoybnzsl1lquf89ej
4911abc53cf0 grafana/grafana:latest "/run.sh" 29 minutes ago Up 29 minutes 3000/tcp tech_grafana.1.oyoot8zkxl92yi05s1ac7nano
a00639d4aa76 eclipse-mosquitto:latest "/docker-entrypoint.…" 29 minutes ago Up 29 minutes 1883/tcp tech_mosquitto.1.toob1isapbmy0h0i4wa97uqq6
ca60cb673379 nginx:latest "/docker-entrypoint.…" 30 minutes ago Up 30 minutes 80/tcp web_nginx.1.mgyk64ggouqi9pyzgkyknmuk4
Network
docker network create --driver overlay --ingress --subnet=10.11.0.0/16 --gateway=10.11.0.1 ingress
docker inspect network ingress
[
{
"Name": "ingress",
"Id": "9ykfwv0ahhk7a9s8g8a2nh53q",
"Created": "2021-02-07T13:52:09.317600628-05:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.11.0.0/16",
"Gateway": "10.11.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": true,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"4911abc53cf0021e0fbad4392a1666e17d500167589460a77bdbd73bc52a75a1": {
"Name": "tech_grafana.1.oyoot8zkxl92yi05s1ac7nano",
"EndpointID": "c9d609ffe49a7aad7be596f6f36bf9b494af109a4c530fc8b5393d0ff3c16753",
"MacAddress": "02:42:0a:0b:00:08",
"IPv4Address": "10.11.0.8/16",
"IPv6Address": ""
},
"9025b777f04c86742b87308374cda51b11530556040faad623dd5db51f246de4": {
"Name": "management_portainer-ce.1.w910sug8vaormu1pgs62wrj2j",
"EndpointID": "bfd0613488921f39d69f4dc4e45e1f2b1b80b7c41a411040f1c2f7518c06f3c6",
"MacAddress": "02:42:0a:0b:00:0c",
"IPv4Address": "10.11.0.12/16",
"IPv6Address": ""
},
"918705a22e73c0f6ea81b063bd95db964d74cbb0f99b5fe19bbfb0cb352b0bfc": {
"Name": "tech_influxdb.1.4d47iflbvoybnzsl1lquf89ej",
"EndpointID": "07a5cac92d0c5fc53e93bf6c049f7b8977dd31c1a2d6fc27a033f1769370c53b",
"MacAddress": "02:42:0a:0b:00:0a",
"IPv4Address": "10.11.0.10/16",
"IPv6Address": ""
},
"a00639d4aa76485b305b41e4052fe955eef4ee2c0f0e90722097d6a7995ca9bc": {
"Name": "tech_mosquitto.1.toob1isapbmy0h0i4wa97uqq6",
"EndpointID": "080d882f2cc09a011261a11c6f46d8c04f26a2329d00f5fa530133bf67a4d8c0",
"MacAddress": "02:42:0a:0b:00:06",
"IPv4Address": "10.11.0.6/16",
"IPv6Address": ""
},
"ca60cb6733797cd6867fb7146217f92ad18d1f759c86dbb65f0804199129651a": {
"Name": "web_nginx.1.mgyk64ggouqi9pyzgkyknmuk4",
"EndpointID": "019506a9da152f923600580121c373e7a74ef99cd05bf3c5094ab92dfe39a472",
"MacAddress": "02:42:0a:0b:00:03",
"IPv4Address": "10.11.0.3/16",
"IPv6Address": ""
},
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "b89f39cca8cfc6553385247315adaf543900a4e56ca769c2989e6ad63baa0ebc",
"MacAddress": "02:42:0a:0b:00:04",
"IPv4Address": "10.11.0.4/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4104"
},
"Labels": {},
"Peers": [
{
"Name": "933a5c2c1e76",
"IP": "192.168.1.103"
}
]
}
]
Inter container in same stack works, but across stack fails.
#from tech stack->web stack fails, inside works
docker exec -it 918705a22e73 bash
root@918705a22e73:/# ping nginx
ping: nginx: Name or service not known
root@918705a22e73:/# ping grafana
PING grafana (10.0.9.5) 56(84) bytes of data.
64 bytes from 10.0.9.5 (10.0.9.5): icmp_seq=1 ttl=64 time=0.099 ms
^C
--- grafana ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.099/0.099/0.099/0.000 ms
#from web->web stack works, web->tech fails
root@ca60cb673379:/# curl grafana
curl: (6) Could not resolve host: grafana
root@ca60cb673379:/# curl nginx
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
root@ca60cb673379:/# curl influxdb
curl: (6) Could not resolve host: influxdb
root@ca60cb673379:/#
Soon I figured (the hard way) default overlay network connected does not offer interstack communication. For that to work you will need to create a new overlay network and associate the services to it.
#create shared network for stacks
docker network create --driver=overlay --attachable shared_overlay
networks:
- default
- shared_overlay