ssldocker-composessl-certificatetraefikacme

traefik acme http challenge yields "No default certificate, fallback to the internal generated certificate"

I have a basic docker compose file with two services: traefik and nginx

services:

  traefik:
    image: traefik:v3.2
    command:
    # - "--api.debug=true"
    - "--log.level=DEBUG"
    - "--api.insecure=true"
    - "--providers.file.directory=/etc/traefik"
    - "--providers.file.watch=true"
    - "--entryPoints.web.address=:80"
    - "--entryPoints.websecure.address=:443"
    - "--certificatesresolvers.myresolver.acme.email=my@email.com"
    - "--certificatesresolvers.myresolver.acme.storage=/etc/traefik/acme.json"
    - "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
    - "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
    # - "--serverstransport.insecureskipverify"
    depends_on:
    - nginx
    volumes:
      - ./traefik:/etc/traefik
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    restart: "unless-stopped"

  nginx:
    image: nginx:latest
    ports:
      - "80"
    volumes:
      - ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/conf/conf.d:/etc/nginx/conf.d
      - ./nginx/data:/usr/share/nginx
    restart: "unless-stopped"

my traefik dynamic config is the following:

http:
  routers:
    nginx:
      entrypoints:
      - web
      - websecure
      rule: "Host(`www.host1.com`) || Host(`www.host2.com`) || Host(`host1.com`) || Host(`host2.com`)"
      service: nginx
  middlewares:
    redirect-to-https:
      redirectScheme:
        permanent: true
        scheme: https
  services:
    nginx:
      loadBalancer:
        servers:
        - url: http://nginx/

When i spin up the compose environment, I see an empty acme.json file created under ./traefik. I cannot make it generate a certificate (here using the staging acme server, but same happens with prod).

root@cloud:~/cloud# docker compose up
[+] Running 3/1
 ✔ Network cloud_default      Created                                                                                                                                                0.1s
 ✔ Container cloud-nginx-1    Created                                                                                                                                                0.0s
 ✔ Container cloud-traefik-1  Created                                                                                                                                                0.0s
Attaching to nginx-1, traefik-1
nginx-1    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx-1    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx-1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx-1    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx-1    | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
nginx-1    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
nginx-1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx-1    | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: using the "epoll" event method
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: nginx/1.25.4
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: OS: Linux 5.15.0-125-generic
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: start worker processes
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: start worker process 29
nginx-1    | 2024/11/11 12:00:56 [notice] 1#1: start worker process 30
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:103 > Traefik version 3.2.0 built on 2024-10-28T14:49:00Z version=3.2.0
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:110 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"insecure":true},"certificatesResolvers":{"myresolver":{"acme":{"caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"email":"my@email.com","httpChallenge":{"entryPoint":"web"},"keyType":"RSA4096","storage":"/etc/traefik/acme.json"}}},"entryPoints":{"traefik":{"address":":8080","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"web":{"address":":80","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{"checkNewVersion":true},"log":{"format":"common","level":"DEBUG"},"providers":{"file":{"directory":"/etc/traefik","watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:626 >
traefik-1  | Stats collection is disabled.
traefik-1  | Help us improve Traefik by turning this feature on :)
traefik-1  | More details on: https://doc.traefik.io/traefik/contributing/data-collection/
traefik-1  |
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator aggregator.ProviderAggregator
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=traefik
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"directory":"/etc/traefik","watch":true}
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/.gitignore
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/acme.json
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /etc/traefik/base.yml
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"myresolver","TLSChallengeProvider":{},"caServer":"https://acme-staging-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"email":"my@email.com","httpChallenge":{"entryPoint":"web"},"keyType":"RSA4096","storage":"/etc/traefik/acme.json","store":{}}
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:221 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=myresolver.acme
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:859 > Testing certificate renew... acmeCA=https://acme-staging-v02.api.letsencrypt.org/directory providerName=myresolver.acme
traefik-1  | 2024-11-11T12:00:56Z INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"redirect-to-https":{"redirectScheme":{"permanent":true,"scheme":"https"}}},"routers":{"nginx":{"entryPoints":["web","websecure"],"rule":"Host(`www.host1.com`) || Host(`www.host2.com`) || Host(`host1.com`) || Host(`host2.com`)","service":"nginx"}},"services":{"nginx":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://nginx/"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=file
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"dashboard_redirect":{"redirectRegex":{"permanent":true,"regex":"^(http:\\/\\/(\\[[\\w:.]+\\]|[\\w\\._-]+)(:\\d+)?)\\/$","replacement":"${1}/dashboard/"}},"dashboard_stripprefix":{"stripPrefix":{"prefixes":["/dashboard/","/dashboard"]}}},"routers":{"acme-http":{"entryPoints":["web"],"priority":9223372036854775807,"rule":"PathPrefix(`/.well-known/acme-challenge/`)","ruleSyntax":"v3","service":"acme-http@internal"},"api":{"entryPoints":["traefik"],"priority":9223372036854775806,"rule":"PathPrefix(`/api`)","ruleSyntax":"v3","service":"api@internal"},"dashboard":{"entryPoints":["traefik"],"middlewares":["dashboard_redirect@internal","dashboard_stripprefix@internal"],"priority":9223372036854775805,"rule":"PathPrefix(`/`)","ruleSyntax":"v3","service":"dashboard@internal"}},"serversTransports":{"default":{"maxIdleConnsPerHost":200}},"services":{"acme-http":{},"api":{},"dashboard":{},"noop":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=myresolver.acme
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:299 > Creating load-balancer entryPointName=web routerName=nginx@file serviceName=nginx@file
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:336 > Creating server entryPointName=web routerName=nginx@file serverName=70a994c85944ea66 serviceName=nginx@file target=http://nginx/
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:299 > Creating load-balancer entryPointName=websecure routerName=nginx@file serviceName=nginx@file
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:336 > Creating server entryPointName=websecure routerName=nginx@file serverName=70a994c85944ea66 serviceName=nginx@file target=http://nginx/
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/stripprefix/strip_prefix.go:32 > Creating middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal middlewareType=StripPrefix routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_stripprefix@internal routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:17 > Creating middleware entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_regex.go:18 > Setting up redirection from ^(http:\/\/(\[[\w:.]+\]|[\w\._-]+)(:\d+)?)\/$ to ${1}/dashboard/ entryPointName=traefik middlewareName=dashboard_redirect@internal middlewareType=RedirectRegex routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/observability/middleware.go:33 > Adding tracing to middleware entryPointName=traefik middlewareName=dashboard_redirect@internal routerName=dashboard@internal
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery

DNS resolution works correctly for all the domains used in the config: i.e. all points to the same IP address, that of the server where this is deployed.

Why isn't traefik receiving the certificate?

traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=myresolver.acme
traefik-1  | 2024-11-11T12:00:56Z DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
Solution

  • You need to define the certificate resolver also in the dynamic config for each router, like this:

    http:
  routers:
    nginx:
      entrypoints:
      - web
      - websecure
      rule: "Host(`www.host1.com`) || Host(`www.host2.com`) || Host(`host1.com`) || Host(`host2.com`)"
      service: nginx
      tls:
        certResolver: myresolver

    Or alternatively in the service labels in docker compose:

      nginx:
    image: nginx:latest
    ...
    labels:
      - traefik.http.routers.nginx.tls=true
      - traefik.http.routers.nginx.tls.certresolver=myresolver

    but since you already do most of the stuff in the dynamic config, having it defined there would make more sense.

    Be sure to revisit the traefik documentation example.