DNS not working within docker containers when host uses dnsmasq and Google's DNS server are firewalled?

The symptom is: the host machine has proper network access, but programs running within containers can't resolve DNS names (which may appear to be "can't access the network" before investigating more).

$ sudo docker run -ti mmoy/ubuntu-netutils /bin/bash
root@082bd4ead733:/# ping www.example.com
... nothing happens (timeout) ... ^C
root@082bd4ead733:/# host www.example.com
... nothing happens (timeout) ... ^C

(The docker image mmoy/ubuntu-netutils is a simple image based on Ubuntu with ping and host included, convenient here since the network is broken and we can't apt install these tools)

The issue comes from the fact that docker automatically configured Google's public DNS as DNS server within the container:

root@082bd4ead733:/# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 8.8.8.8
nameserver 8.8.4.4

This just works in many configurations, but obviously doesn't when the host runs on a network where Google's public DNS are filtered by some firewall rules.

The reason this happened is:

Docker first tries configuring the same DNS server(s) on the host and within the container.
The host runs dnsmasq, a DNS caching service. dnsmasq acts as a proxy for DNS requests, hence the apparent DNS server in the host's /etc/resolve.conf is nameserver 127.0.1.1, i.e. localhost.
The host's dnsmasq listens only for requests comming from localhost and blocks requests coming from the docker container.
Since using 127.0.1.1 within docker doesn't work, docker falls back to Google's public DNS, which do not work either.

There may be several reasons why DNS is broken within docker containers. This question (and answers) covers the case where:

dnsmasq is used. To check whether this is the case:
- Run ps -e | grep dnsmasq on the host. If the output is empty, you're not running dnsmasq.
- Check the host's resolv.conf, it probably contains an entry like nameserver 127.0.1.1. If it contains nameserver 127.0.0.53, you're probably running systemd-resolved instead of dnsmasq. If so, you won't be able to use the solution forwading DNS requests to dnsmasq (the one using listen-address=172.17.0.1). systemd-resolved versions earlier than 247 hardcoded the fact that it listens only on the 'lo' interface hence there's no easy way to adapt this solution with these versions. Other answers below will work with systemd-resolved.
Google's public DNS is filtered. Run host www.example.com 8.8.8.8. If it fails or times out, then you are in this situation.

What are the solutions to get a proper DNS configuration in this configuration?

Solution

A clean solution is to configure docker+dnsmasq so than DNS requests from the docker container are forwarded to the dnsmasq daemon running on the host.

For that, you need to configure dnsmasq to listen to the network interface used by docker, by adding a file /etc/NetworkManager/dnsmasq.d/docker-bridge.conf:

$ cat /etc/NetworkManager/dnsmasq.d/docker-bridge.conf
listen-address=172.17.0.1

Then restart network manager to have the configuration file taken into account:

sudo service network-manager restart

Once this is done, you can add 172.17.0.1, i.e. the host's IP address from within docker, to the list of DNS servers. This can be done either using the command-line:

$ sudo docker run -ti --dns 172.17.0.1 mmoy/ubuntu-netutils bash
root@7805c7d153cc:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.6 ms

... or through docker's configuration file /etc/docker/daemon.json (create it if it doesn't exist):

$ cat /etc/docker/daemon.json                      
{
  "dns": [
    "172.17.0.1",
        "8.8.8.8",
        "8.8.4.4"
  ]
}

(this will fall back to Google's public DNS if dnsmasq fails)

You need to restart docker to have the configuration file taken into account:

sudo service docker restart

Then you can use docker as usual:

$ sudo docker run -ti mmoy/ubuntu-netutils bash
root@344a983908cb:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.3 ms