docker pull tries to talk http to registry that only understands https

I have an ordinary ubuntu image with no dockerd installed, only the docker command line client and curl such that I can query a docker registry. I have set up tls security. I use a third container for the docker daemon (dind). The following works as expected.

curl https://myregistry/v2/_catalog # gets the info
curl http://myregistry/v2/_catalog  # fails as expected

So I expect that docker can pull from the registry via https. Unfortunately my docker client only tries http, which of course fails.

docker pull myregistry/myimage
Using default tag: latest
Error response from daemon: Client sent an HTTP request to an HTTPS server.

My question is how to tell docker pull to use https instead of http? All posts I found so far describe the opposite problem where docker talks https to an http registry.

I assume that my certificates are correct, because otherwise curl would have failed with both http and https.

Update One error was as BMich pointed out, that in myregistry/image the myregistry is a namespace and not my hostname. So I renamed it to my.hub. The dot in between makes this a hostname.

I have not set insecure registry on any os or container and also not in any config.toml of a gitlab-runner.

Minimal Reproducible Example I have added following hosts to /etc/hosts: myregistry, dind, client1, client2.

#!/bin/bash

docker stop my.hub dind client1 client2
docker rm my.hub dind client1 client2
sudo rm -rf volumes

CURRENT=${PWD}
HUB_DIR=${PWD}/volumes/my.hub
DIND_DIR=${PWD}/volumes/dind

sudo rm -rf $CURRENT/volumes
mkdir -p $CURRENT/ca
mkdir -p $HUB_DIR/certs
mkdir -p $DIND_DIR/certs/ca
mkdir -p $DIND_DIR/certs/client
mkdir -p $DIND_DIR/certs/server
mkdir -p $DIND_DIR/etc/docker/certs.d/my.hub
mkdir -p $DIND_DIR/etc/docker/certs.d/my.hub:443
mkdir -p $DIND_DIR/usr/local/share/ca-certificates

# CA Key/Certificate key.pem and cert.pem
cd $CURRENT/ca

openssl genrsa -out key.pem 2048
openssl req -x509 -days 365 -new -nodes -key key.pem \
    -subj "/C=UK/ST=Sussex/L=London/O=Moiself/OU=Sofa/CN=myregistry" \
    -sha256 -out cert.pem

# my.hub key/cert pair
cd $HUB_DIR/certs

openssl genrsa -out key.pem 2048
openssl req -new -key key.pem -out csr.pem \
    -subj "/C=UK/ST=Sussex/L=London/O=Moiself/OU=Sofa/CN=myregistry" \
    -addext "subjectAltName=DNS:my.hub,DNS:localhost"
openssl x509 -req -days 365 -sha256 -in csr.pem  \
    -CA $CURRENT/ca/cert.pem -CAkey $CURRENT/ca/key.pem \
    -CAcreateserial -out cert.pem \
    -extfile <(printf "authorityKeyIdentifier=keyid,issuer\nbasicConstraints=CA:FALSE\nkeyUsage=critical,digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment\nsubjectAltName=DNS:my.hub,DNS:localhost")

cd $CURRENT

cp $CURRENT/ca/cert.pem    $DIND_DIR/certs/client/ca.pem
cp $CURRENT/ca/cert.pem    $DIND_DIR/certs/ca/cert.pem
cp $CURRENT/ca/key.pem     $DIND_DIR/certs/ca/key.pem
cp $CURRENT/ca/cert.pem    $DIND_DIR/usr/local/share/ca-certificates/ca.crt
cp $HUB_DIR/certs/cert.pem $DIND_DIR/etc/docker/certs.d/my.hub/ca.crt

sudo chown -R root volumes

docker run -d -it --network some-network --ip 172.18.0.2 -e REGISTRY_HTTP_ADDR=0.0.0.0:443  \
    -e REGISTRY_HTTP_TLS_KEY=/certs/key.pem -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/cert.pem \
    -v $HUB_DIR/certs:/certs:ro --name my.hub --hostname my.hub registry:2

docker run -d -it --privileged --network some-network --ip 172.18.0.3 --network-alias docker \
    -e DOCKER_TLS_CERTDIR=/certs  \
    -v $DIND_DIR/etc/docker/certs.d:/etc/docker/certs.d:ro \
    -v $DIND_DIR/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro  \
    -v $DIND_DIR/certs/client:/certs/client  \
    -v $DIND_DIR/certs/ca:/certs/ca  \
    -v $DIND_DIR/certs/server:/certs/server \
    --name dind --hostname dind docker:dind  \
    sh -c "dockerd-entrypoint.sh --iptables=false && tail -f /dev/null"

sleep 3
docker exec -it dind apk update
docker exec -it dind apk add curl
docker exec -it dind /usr/sbin/update-ca-certificates
docker exec -it dind docker pull alpine:latest
docker exec -it dind docker tag alpine:latest my.hub/alpine2
docker exec -it dind docker push my.hub/alpine2
docker exec -it dind curl https://my.hub/v2/_catalog
docker exec -it dind docker images

sudo rm -rf $DIND_DIR/certs/client2
sudo cp -r  $DIND_DIR/certs/client $DIND_DIR/certs/client2
cp $DIND_DIR/certs/client/cert.pem $DIND_DIR/etc/docker/certs.d/dind/ca.crt
docker run  -d -it --network some-network --ip 172.18.0.4 \
    -e DOCKER_HOST=tcp://dind:2376 \
    -e DOCKER_TLS_CERTDIR=/certs \
    -v $DIND_DIR/certs/client2:/certs:ro \
    -v $DIND_DIR/etc/docker/certs.d:/etc/docker/certs.d:ro \
    -v $DIND_DIR/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro  \
    --name client1  --hostname client1 docker:24.0.0-beta.1-cli-alpine3.17

sleep 3
docker exec -it client1 apk update
docker exec -it client1 apk add curl
docker exec -it client1 /usr/sbin/update-ca-certificates
docker exec -it client1 docker pull alpine:latest;              # Sending HTTP error
docker exec -it client1 docker tag alpine:latest my.hub/alpine3 # Sending HTTP error
docker exec -it client1 docker push my.hub/alpine3              # Sending HTTP error
docker exec -it client1 curl https://my.hub/v2/_catalog
docker exec -it client1 docker images                           # Sending HTTP error

docker run  -d -it --network some-network  --ip 172.18.0.5  \
    -e DOCKER_HOST=tcp://dind:2376  \
    -e DOCKER_TLS_CERTDIR=/certs \
    -v $DIND_DIR/etc/docker/certs.d:/etc/docker/certs.d:ro \
    -v $DIND_DIR/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro  \
    --name client2  --hostname client2 localci-ubuntu-gcc

sleep 3
docker exec -it client2 apt update
docker exec -it client2 /usr/sbin/update-ca-certificates
docker exec -it client2 docker pull alpine:latest               # Sending HTTP error
docker exec -it client2 docker tag alpine:latest my.hub/alpine4 # Sending HTTP error
docker exec -it client2 docker push my.hub/alpine4              # Sending HTTP error
docker exec -it client2 curl https://my.hub/v2/_catalog
docker exec -it client2 docker images                           # Sending HTTP error

Docker talks to the registry with http when the registry is configured as an "insecure registry". That setting should be visible in docker info, and would commonly be configured in /etc/docker/daemon.json (changes apply when reloading or restarting the docker engine). Remove your registry from this configuration to return to the default TLS connection.

For more details, see https://docs.docker.com/reference/cli/dockerd/#insecure-registries

For a full running example creating the certifications, running the registry with the registry:2 image, and running the docker engine with DinD, here's a script:

#!/bin/sh

set -e

# setup TLS for CA
[ -f ca-key.pem ] || openssl genrsa -out ca-key.pem 4096
openssl req -subj "/CN=CA" -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem </dev/null
# setup TLS cert for registry
[ -f registry-key.pem ] || openssl genrsa -out registry-key.pem 2048
openssl req -subj "/CN=registry.localdomain" -new -key registry-key.pem -out registry.csr </dev/null
echo "subjectAltName = DNS:registry.localdomain" >registry.cnf
openssl x509 -req -days 365 -in registry.csr -CA ca.pem -CAkey ca-key.pem \
  -CAcreateserial -out registry-cert.pem -extfile registry.cnf </dev/null
cat registry-cert.pem ca.pem >registry-chain.pem
rm registry.cnf registry.csr

# cleanup old containers
for container in registry-test registry-dind-server registry-dind-client; do
  docker inspect ${container} >/dev/null 2>&1 \
  && docker stop ${container} && docker rm ${container} || true
done

# create network
docker network create registry-test >/dev/null 2>&1 || true

# start registry
docker run --name registry-test -d --rm \
  --network registry-test --network-alias registry.localdomain \
  -e "REGISTRY_HTTP_ADDR=:443" \
  -e "REGISTRY_HTTP_HOST=https://registry.localdomain" \
  -e "REGISTRY_HTTP_TLS_CERTIFICATE=/certs/cert.pem" \
  -e "REGISTRY_HTTP_TLS_KEY=/certs/key.pem" \
  -e "REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry" \
  -v "registry-data:/var/lib/registry" \
  -v "$(pwd)/registry-chain.pem:/certs/cert.pem:ro" \
  -v "$(pwd)/registry-key.pem:/certs/key.pem" \
  registry:2

# start DinD server
docker run --name registry-dind-server -d --rm --privileged \
  --network registry-test --network-alias docker \
  -e "DOCKER_TLS_CERTDIR=/certs" \
  -v "registry-dind-certs:/certs/client" \
  -v "registry-dind:/var/lib/docker" \
  -v "$(pwd)/registry-chain.pem:/etc/docker/certs.d/registry.localdomain/ca.crt:ro" \
  docker:dind

# run the client (note DOCKER_TLS_VERIFY and DOCKER_CERT_PATH are explicitly set to bypass the normal entrypoint checks)
docker run --name registry-dind-client -it --rm \
  --network registry-test --network-alias docker-client \
  -e "DOCKER_HOST=tcp://docker:2376" \
  -e "DOCKER_CERT_PATH=/certs" \
  -e "DOCKER_TLS_VERIFY=1" \
  -v "registry-dind-certs:/certs:ro" \
  docker:latest sh

# cleanup
for container in registry-test registry-dind-server; do
  docker inspect ${container} >/dev/null 2>&1 \
  && docker stop ${container} || true
done
docker network rm registry-test || true
# optional data cleanup removes all pulled images in the DinD server and registry content
sleep 2
docker volume rm registry-dind-certs registry-dind registry-data || true

And from in that script, you would access the registry like:

/ # docker pull alpine
Using default tag: latest
latest: Pulling from library/alpine
da9db072f522: Pull complete
Digest: sha256:1e42bbe2508154c9126d48c2b8a75420c3544343bf86fd041fb7527e017a4b4a
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest

/ # docker tag alpine registry.localdomain/library/alpine

/ # docker push registry.localdomain/library/alpine
Using default tag: latest
The push refers to repository [registry.localdomain/library/alpine]
75654b8eeebd: Pushed
latest: digest: sha256:3e21c52835bab96cbecb471e3c3eb0e8a012b91ba2f0b934bd0b5394cd570b9f size: 527

/ # exit

As an addendum, I believe this error:

docker pull myregistry/myimage
Using default tag: latest
Error response from daemon: Client sent an HTTP request to an HTTPS server.

is actually because the DinD client started before the DinD server was running, so the DOCKER_TLS_VERIFY value never got set by the entrypoint script. This has nothing to do with the registry configuration, or setting the certificate in the docker engine, every single docker command would fail with the same error trying to talk to the docker server. If you are running in automation, you'll want to add some checks and delay loops to wait for the server to fully start.