I'm stuck with a problem.
I have four hosts (1, 2, 3, 4) inside Rancher 1.6 UI. All hosts have almost the same configuration: nginx, PHP, MongoDB.
Host 1 has both rancher/server and rancher/agent up and running. Hosts 2, 3 and 4 have only rancher/agent.
The problem is, all the port forwarding settings for nginx on hosts 2, 3 and 4 are completely ignored: all of the requests to 80 and 443 ports are forwarded to Host 1 and I have no idea how to overcome that.
When I execute command iptables -nvL -t nat | grep docker0 on Host 2, 3 or 4, I get the same output like:
104 6565 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE tcp -- * !docker0 10.42.0.0/16 0.0.0.0/0 ! match-set RANCHER_DISABLE_HOST_NAT_IPSET dst masq ports: 1024-65535
0 0 MASQUERADE udp -- * !docker0 10.42.0.0/16 0.0.0.0/0 ! match-set RANCHER_DISABLE_HOST_NAT_IPSET dst masq ports: 1024-65535
0 0 MASQUERADE all -- * !docker0 10.42.0.0/16 0.0.0.0/0 ! match-set RANCHER_DISABLE_HOST_NAT_IPSET dst
4 240 MASQUERADE all -- * docker0 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL dst-type UNICAST
0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 5.67.218.11 tcp dpt:80 to:10.42.204.73:80
0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 5.67.218.11 tcp dpt:443 to:10.42.204.73:443
1 528 DNAT udp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:500 to:10.42.6.75:500
0 0 DNAT udp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:4500 to:10.42.6.75:4500
0 0 DNAT tcp -- !docker0 * 0.0.0.0/0 5.67.218.11 tcp dpt:27077 to:10.42.87.201:27017
152 36328 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
where 5.67.218.11 is the IP address of Host 1.
If I stop Nginx container on Host 1, ports 80 and 443 disappear from the output.
How to make docker (or Rancher) use port forwarding locally on each host and not forward requests to Host 1?
UPD: I have another set of servers with the same configuration, and if I execute command iptables -nvL -t nat | grep docker0 on some of the hosts in that set, I get another output, like:
2078K 132M MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
2405 144K MASQUERADE tcp -- * !docker0 10.42.0.0/16 0.0.0.0/0 ! match-set RANCHER_DISABLE_HOST_NAT_IPSET dst masq ports: 1024-65535
368 194K MASQUERADE udp -- * !docker0 10.42.0.0/16 0.0.0.0/0 ! match-set RANCHER_DISABLE_HOST_NAT_IPSET dst masq ports: 1024-65535
0 0 MASQUERADE all -- * !docker0 10.42.0.0/16 0.0.0.0/0 ! match-set RANCHER_DISABLE_HOST_NAT_IPSET dst
10433 626K MASQUERADE all -- * docker0 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL dst-type UNICAST
14 4046 DNAT udp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:500 to:10.42.216.158:500
89 4639 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:10.42.186.151:80
53 2856 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:27017 to:10.42.155.127:27017
4 1240 DNAT udp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:4500 to:10.42.216.158:4500
58 3144 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:10.42.186.151:443
5907K 1412M RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
The only difference is: instead of some IP address of the Host with rancher/server entries have 0.0.0.0/0, and everything works just fine. But I can't find out how to make another set of servers work this way.
Haven't found the solution of the described problem. The only thing that helped was to move rancher/server to another server instance which solved the problem.
!!! Don't run both rancher/server and rancher/agent on the same environment, since this may result problems with propagation of iptables to other hosts in your infrastracture with rancher/agent installed.