Network Issue: Timeout in Docker vs working in Host : TLS issue leads to not reachable in docker without network=host

Info:

TL;DR:

In case you experience network issues (timeouts, erratic behavior, etc) on docker container while it works on host, or with --network host, this issue is related.
This Issue is solved - please see accepted answer.
It took some iterations to reach there, and I leave these iterations in this issue for seo.
This issue is sometimes reproducable, sometimes not. logs exist in this question.

There are 3 Observations and logs attached below.

Minimal Example:

this fails:

docker run --rm -it alpine /bin/sh
apk update && apk add curl openssl
curl -L https://microsoft.com -v

this works:

docker run --rm -it --network host alpine /bin/sh
apk update && apk add curl openssl
curl -L https://microsoft.com -v

First Analysis

The first Analysis contains the logs show the discrepancy between Container and Host for a website which fails and a website which works.
The second Analysis show the logs for the previously failing website on Container and Host and point out the difference between failing situation and working situation.
Finally at the end there's the Question deriving from the whole situation - how to configure the container to overcome such a v6 issue.

Original Problem:

In Docker Container it's not possible to reach any microsoft website / repository.

I'm trying to add the microsoft repository to different docker container (eg alpine, debian 12) and it is not possible to reach the microsoft repository (or any microsoft website) from within the container, while google works.

With network=host it is possible - but it can't be used because of reasons.

First Analysis:

In Container:

verbose output of curl to microsoft.com and google.com

root@a49cc57027b7:/var/www/html# curl -L https://microsoft.com -v
*   Trying 20.236.44.162:443...
* Connected to microsoft.com (20.236.44.162) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
^C // cancelled after endless waiting


root@a49cc57027b7:/var/www/html# curl -L https://google.com -v
*   Trying 142.250.186.46:443...
* Connected to google.com (142.250.186.46) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
...

In Host

verbose output of curl to microsoft.com and google.com from host

me@me: ~ $ curl -L https://microsoft.com -v
 *   Trying 20.236.44.162:443...
 *   Trying [2603:1020:201:10::10f]:443...
 * Immediate connect fail for 2603:1020:201:10::10f: Network is unreachable
 * Connected to microsoft.com (20.236.44.162) port 443 (#0)
 * ALPN: offers h2,http/1.1
 * TLSv1.3 (OUT), TLS handshake, Client hello (1):
 *  CAfile: /etc/ssl/certs/ca-certificates.crt
 *  CApath: /etc/ssl/certs
 * TLSv1.3 (IN), TLS handshake, Server hello (2):
 * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
 * TLSv1.3 (IN), TLS handshake, Certificate (11):
 * TLSv1.3 (IN), TLS handshake, CERT verify (15):
 * TLSv1.3 (IN), TLS handshake, Finished (20):
 * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
 * TLSv1.3 (OUT), TLS handshake, Finished (20):
 * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
 * ALPN: server accepted h2
 * Server certificate:
 *  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=microsoft.com
 *  start date: Aug  8 18:46:02 2023 GMT
 *  expire date: Jun 27 23:59:59 2024 GMT
 *  subjectAltName: host "microsoft.com" matched cert's "microsoft.com"
 *  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
 *  SSL certificate verify ok.
 * using HTTP/2


me@me: ~ $ curl -L https://google.com -v
*   Trying 142.250.186.46:443...
* Connected to google.com (142.250.186.46) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.google.com
*  start date: Jul 31 08:16:44 2023 GMT
*  expire date: Oct 23 08:16:43 2023 GMT
*  subjectAltName: host "google.com" matched cert's "google.com"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
*  SSL certificate verify ok.
* using HTTP/2

Diff:

microsoft tries to establish v6
different algorithms for tls

Analysis 2

Meanwhile it isn't reproducable anymore - no changes in host, but the logs show that the v6 log on Microsoft side doesn't exist anymore.

Container:

root@b4c666549985:/var/www/html# curl -L https://microsoft.com -v
*   Trying 20.76.201.171:443...
* Connected to microsoft.com (20.76.201.171) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=microsoft.com
*  start date: Aug  8 18:46:02 2023 GMT
*  expire date: Jun 27 23:59:59 2024 GMT
*  subjectAltName: host "microsoft.com" matched cert's "microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
*  SSL certificate verify ok.
* using HTTP/2

Host:

me@me:~ $ curl -L https://microsoft.com -v
*   Trying 20.76.201.171:443...
* Connected to microsoft.com (20.76.201.171) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=microsoft.com
*  start date: Aug  8 18:46:02 2023 GMT
*  expire date: Jun 27 23:59:59 2024 GMT
*  subjectAltName: host "microsoft.com" matched cert's "microsoft.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
*  SSL certificate verify ok.
* using HTTP/2

Analysis 3

Meanwhile it is reproducable again - no changes in host. It doesn't work for IP 20.76.201.171:443 (Analysis 2, worked before)

Container:

/ # openssl s_client -connect 20.76.201.171:443
CONNECTED(00000003)

Host and Container running Host Network Mode:

me@me:~ $ openssl s_client -connect 20.76.201.171:443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
verify return:1
depth=1 C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 01
verify return:1
depth=0 C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.oneroute.microsoft.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = *.oneroute.microsoft.com
   i:C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 01
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA384
   v:NotBefore: May 24 10:48:49 2023 GMT; NotAfter: May 18 10:48:49 2024 GMT
 1 s:C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 01
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA384
   v:NotBefore: Jul 29 12:30:00 2020 GMT; NotAfter: Jun 27 23:59:59 2024 GMT
---
...

Solution

Root cause was MTU on Host Network.
The problematic machine sits in a vpn, and the vpn has 1420 MTU.
This can be assessed with ip link.

ip link | grep wg
wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state ...

the mtu can be set in a network:

# docker-compose.yml
  networks:
    my-net:
      driver: bridge
      driver_opts: {com.docker.network.driver.mtu: 1420}

# docker
docker network create --driver bridge --opt "com.docker.network.driver.mtu=1420" my_network

docker run --rm -it --network my_network alpine /bin/sh
apk update && apk add curl openssl
curl -L https://microsoft.com -v