[SOLVED] Unable to reach network 172.16.0.0/24 from containers on an external node with an IPsec tunnel to vCloud Director

Unable to reach network 172.16.0.0/24 from containers on an external node with an IPsec tunnel to vCloud Director

I have an issue using Docker Swarm with IPsec tunnel.

I have a Swarm cluster with 7 nodes hosted on a vCloud Director instance. All these 7 VMs are on the 172.16.0.0/24 network.

Additionally, I have two external nodes in the cluster that are hosted on dedicated servers outside this network. Both of these nodes have an interface on VLAN 4000, with an IP range of 192.168.99.0/24.

External node #1 has set up an IPsec tunnel to my vCloud Director with the following configurations:

leftsubnet=172.16.0.0/24 rightsubnet=192.168.99.0/24

All nodes in my cluster can communicate with each other using their respective private IPs.

Now, here's where I'm facing a problem:

Containers on my external node #2 can reach IPs on the 172.16.0.0/24 network.
However, containers on my external node #1 (which initiates the IPsec tunnel) can't communicate with the 172.16.0.0/24 network.

Does anyone have any idea what might be causing this issue?

MTR from external node #2 to 172.16.0.0/24:

 Host                                                                                                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.18.0.1                                                                                                                                                    0.0%     5    0.0   0.0   0.0   0.1   0.0
 2. 192.168.99.1                                                                                                                                                  0.0%     5    0.3   0.3   0.3   0.4   0.0
 3. (waiting for reply)
 4. 172.16.0.30                                                                                                                                                   0.0%     5   14.7  15.4  14.7  16.1   0.6

MTR from external node #1 to 172.16.0.0/24:

 Host                                                                                                                                                            Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.18.0.1                                                                                                                                                    0.0%     3    0.1   0.1   0.0   0.1   0.0
 2. static.X.X.X.X.clients.your-server.de                                                                                                                   0.0%     3    0.2   0.3   0.2   0.4   0.1
 3. XXX.fsn1.hetzner.com                                                                                                                                       0.0%     3    0.3   0.5   0.3   0.8   0.3
 4. XXX.nbg1.hetzner.com                                                                                                                                       0.0%     3    2.7   3.8   2.7   6.0   1.9
 5. (waiting for reply)

On the external node #1, I have the following iptables rules specifics to IPsec tunnel, just before the Docker rules:

-A FORWARD -s 172.16.0.0/24 -d 192.168.99.0/24 -i enp35s0 -m policy --dir in --pol ipsec --reqid 1 --proto esp -j ACCEPT
-A FORWARD -s 192.168.99.0/24 -d 172.16.0.0/24 -o enp35s0 -m policy --dir out --pol ipsec --reqid 1 --proto esp -j ACCEPT
-A FORWARD -s 172.16.0.0/24 -d 192.168.99.0/24 -i enp35s0 -m policy --dir in --pol ipsec --reqid 1 --proto esp -j ACCEPT
-A FORWARD -s 192.168.99.0/24 -d 172.16.0.0/24 -o enp35s0 -m policy --dir out --pol ipsec --reqid 1 --proto esp -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-INGRESS
-A FORWARD -j DOCKER-ISOLATION-STAGE-1

Solution

You could use a SNAT rule on the Docker host with the IPsec tunnel:

iptables -t nat -I POSTROUTING 1 -s 172.18.0.0/24 -d 172.16.0.0/24 -o enp35s0 -j SNAT --to-source 192.168.99.X

Don't forget to change 192.168.99.X by the IP used by IPsec on your Docker host.