I have a on AWS EC2 Instances Docker Containers running, the Docker Containers are Connected over a Docker Network which is replicated over a etcd cluster. But all of the Containers can't connect outside. For example if I ping or curl a host get a 100% packages lost. The etcd network communicates over the AWS Private IP.
Here is my docker info:
root@ip-10-0-127-34:/home# docker info
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 9
Server Version: 17.05.0-ce
Storage Driver: devicemapper
Pool Name: docker-202:1-2050903-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 4.554GB
Data Space Total: 107.4GB
Data Space Available: 102.8GB
Metadata Space Used:sattler 6.922MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.141GB
Thin Pool Minimum Free Space: 3.221GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1028-aws
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.303GiB
Name: ip-10-0-127-34
ID: JFQO:C56I:VM22:UDKJ:QDKD:HUDT:KL3X:JCSR:WAPG:66JL:S4RM:4ENN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Cluster Store: etcd://127.0.0.1:2379
Cluster Advertise: 10.0.127.34:2376
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: No swap limit support
Docker Inspect the Network:
{
"Name": "df-global",
"Id": "59aef8ccc5d7464ee715428783b9b03ba7737d298f41cc30a21f4856e75b92c1",
"Created": "2017-08-14T09:51:31.650109966Z",
"Scope": "global",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "192.168.1.0/14"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"Containers": {
"ep-f257d697653d801395f9154d086a6290694d6ded7aefe827c67dcd10808023eb": {
"Name": "elasticsearch-data-1",
"EndpointID": "f257d697653d801395f9154d086a6290694d6ded7aefe827c67dcd10808023eb",
"MacAddress": "02:42:c0:a8:00:0b",
"IPv4Address": "192.168.0.11/14",
"IPv6Address": ""
},
"ep-f5c1be7a768802154b58b33e67a3f009df1d4f1336297df9f914be9a720bd8ce": {
"Name": "postgresql-vg_internal01-prod",
"EndpointID": "f5c1be7a768802154b58b33e67a3f009df1d4f1336297df9f914be9a720bd8ce",
"MacAddress": "02:42:c0:a8:00:02",
"IPv4Address": "192.168.0.2/14",
"IPv6Address": ""
The same problem is described here: https://forums.docker.com/t/ping-between-containers-on-different-docker-host-but-connected-by-overlay-networ-fails/9960 but there is no answer
I was still able to ping 172.17.2.1 and yet that was the only ip address I could contact from within a container. Turns out someone/thing cleared out the iptables firewall and had every chain in every table replaced with default ACCEPT rules! All the docker nat and forwarding rules were missing. A quick reboot solved the issue.
A temporary solution was to use the docker command line option --network 'host'.