dockernetwork-programmingroutestraefik

traefik v2 can't curl from inside via domain to other container of my docker

I try to accomplish the following thing: From Service B I want to call Service A

Working:

Not working (No feedback from curl, no error):

So I did try to debug this, but curl and traefik dont show logs, where the catch is. CURL is just hanging.

nslookup of sub.example.org => ok
ping of sub.example.org => ok
curl of sub.example.org => hangs not output beside: connecting to ip

All external domains/routes are SSL encrpyted

enter image description here

Solution

  • Given that DNS and ping work, this most likely is a firewall issue. DNS probably does not resolve to the private ip of the container that was assigned by docker, but to the host ip. The firewall must allow traffic from the docker bridge network to the host ip on port 443. But when mapping container ports (443:443) docker will only configure the firewall to allow traffic from outside the bridge network to the private container ip in the bridged network - which is not enough to support the desired use case. So, let's understand firewalls again and fix it! :)

    TL;DR

    Allow https traffic from Service B ip/network to the host ip, for example using ufw:

    sudo ufw allow from <service-B-ip or docker-bridge-network> to <host-ip> port 443

    You can find the subnet of the docker bridge network using docker network inspect <network-name> | grep Subnet, e.g. 172.18.0.0/16. Assuming the host ip is 10.0.0.4 this would allow all services on the bridged network to communicate to 443 (traefik):

    sudo ufw allow from 172.18.0.0/16 to 10.0.0.4 port 443

    Also make sure ufw is enabled: sudo ufw enable.

    Understand docker networking and firewalls

    Firewalls test where a request is going (network interface, target IP, port) and where it comes from (network interface, ip) and some other stuff. So let's check what rules we need to allow the traffic desired in the use case and then check if the firewall is configured accordingly.

    Run docker network ls and you will find a network of type bridge with some id for your network, e.g. cd0af4596aee. You can also check ifconfig and will find that this network interface was created by docker once you created the network (br-cd0af4596aee).

    When mapping a port of a container to a host port (e.g. 443:443 for traefik) docker sets some firewall rules. Docker does so by manipulating iptables on linux based systems, let's check what docker does:

    $ sudo iptables -L -v -n | grep 443
pkts bytes target     prot opt in               out              source               destination
  13   780 ACCEPT     tcp  --  !br-cd0af4596aee br-cd0af4596aee  0.0.0.0/0            172.18.0.4           tcp dpt:443

    The rule allows any source but only via interfaces that are not the bridge interface of the network (!br-cd0af4596aee) and only to the private ip 172.18.0.4. But the traffic we want to allow originates from the network br-cd0af4596aee and targets the host ip to which DNS resolves (which probably is not the private ip of the container that was assigned by docker).

    Understanding this, you probably now understand what a solution will look like: We need to allow traffic from the network itself to the host ip. Because it is cumbersome to hassle with the network interfaces, we can also add a rule that just allows all in and out interfaces but only for specific ips/networks. For this we can use ufw which is available on most ubuntu distributions. ufw is a frontend to iptables that is much easier to use than iptables. Note that docker rules in iptables live before ufw rules - effectively making docker bypassing ufw rules that try to block specific traffic. See here for more information.

    Assuming 172.18.0.0/16 is the ip range of the network and 10.0.0.4 the host ip:

    # sudo ufw allow from <service-B-ip or network-of-the-service> to <host-ip> port 443
sudo ufw allow from 172.18.0.0/16 to 10.0.0.4 port 443

    Check iptables again:

    $ sudo iptables -L -v -n | grep 443
13   780 ACCEPT     tcp  --  !br-cd0af4596aee br-cd0af4596aee  0.0.0.0/0            172.18.0.4           tcp dpt:443
1    60 ACCEPT     tcp  --  *      *       172.18.0.0/16        0.0.0.0/0            tcp dpt:443
0     0 ACCEPT     udp  --  *      *       172.18.0.0/16        0.0.0.0/0            udp dpt:443

    This should fix the issue.

    But why do DNS and ping work?