Traefik can't connect to ACME Server
I have a portainer and traefik infrastructure on an ionos VPS. Suddenly overnight it broke. I can't access any dashboard or service on the server.
Investigating further led me to believe it probably is an issue with the certificate generation but I can't say for sure, if it's the only problem.
So I haven't touched it in a few months and now it's not working anymore. To be clear the docker-compose.yml in this post is slightly edited, I changed from latest for traefik to v3. Other than that it's the exact same as the one that has been running for over half a year.
So what have I tried already:
- Searching for a solution obviously, no luck.
- Reaching the acme server from the vps host, is working.
- Reaching the acme server from inside the docker container not working.
- Rebooting the whole machine no change.
- Reloading the docker service no change.
- Purging and re-adding networks no change.
If anyone has any ideas on what the problem might be, I'm very open for ideas or solutions.
services:
traefik:
container_name: traefik
image: "traefik:v3"
healthcheck:
test: ["CMD", "traefik", "healthcheck", "--ping"]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
restart: always
command:
- --providers.docker
- --providers.docker.exposedbydefault=false
- --providers.docker.network=traefik-proxy
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --api.dashboard=true
- --ping=true
- --providers.docker
- --log.level=ERROR
- --certificatesresolvers.leresolver.acme.httpchallenge=true
- --certificatesresolvers.leresolver.acme.email=notyourproblem
- --certificatesresolvers.leresolver.acme.storage=./acme.json
- --certificatesresolvers.leresolver.acme.httpchallenge.entrypoint=web
ports:
- "80:80"
- "443:443"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./acme.json:/acme.json"
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-proxy"
- "traefik.http.routers.dashboard.rule=Host(`nope`) && (PathPrefix(`/shush`) || PathPrefix(`/nope`))"
- "traefik.http.routers.dashboard.entrypoints=websecure"
- "traefik.http.routers.dashboard.service=api@internal"
- "traefik.http.routers.dashboard.middlewares=auth"
- "traefik.http.services.dashboard.loadbalancer.server.port=8080"
- "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
- "traefik.http.routers.http-catchall.entrypoints=web"
- "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
- "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
- "traefik.http.middlewares.auth.basicauth.users=td3v:noneofyourconcern"
- "traefik.http.routers.dashboard.tls.certresolver=leresolver"
networks:
- traefik-proxy
portainer:
container_name: portainer
image: portainer/portainer-ce:latest
command: -H unix:///var/run/docker.sock
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- portainer_data:/data
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-proxy"
- "traefik.http.routers.frontend.rule=Host(`nope`)"
- "traefik.http.routers.frontend.entrypoints=websecure"
- "traefik.http.services.frontend.loadbalancer.server.port=9000"
- "traefik.http.routers.frontend.service=frontend"
- "traefik.http.routers.frontend.tls.certresolver=leresolver"
- "traefik.http.routers.edge.rule=Host(`nope`)"
- "traefik.http.routers.edge.entrypoints=websecure"
- "traefik.http.services.edge.loadbalancer.server.port=8000"
- "traefik.http.routers.edge.service=edge"
- "traefik.http.routers.edge.tls.certresolver=leresolver"
networks:
- traefik-proxy
- internal
volumes:
portainer_data:
networks:
traefik-proxy:
external: true
internal:
external: false
Network list of docker
Firewall rules in the ionos panel
logs:
Creating network "portainer-traefik_internal" with the default driver
Creating traefik ... done
Creating portainer ... done
Attaching to traefik, portainer
portainer | 2024/11/07 03:37PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
portainer | 2024/11/07 03:37PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
portainer | 2024/11/07 03:37PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer | 2024/11/07 03:37PM INF github.com/portainer/portainer/api/chisel/service.go:198 > Found Chisel private key file on disk | private-key=/data/chisel/private-key.pem
portainer | 2024/11/07 15:37:44 server: Reverse tunnelling enabled
portainer | 2024/11/07 15:37:44 server: Fingerprint IShTAt+SYsnH0uQO8hqboTUl+fzRKtZT3mgqg33mF3k=
portainer | 2024/11/07 15:37:44 server: Listening on http://0.0.0.0:8000
portainer | 2024/11/07 03:37PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:649 > starting Portainer | build_number=35428 go_version=1.20.5 image_tag=linux-amd64-2.19.4 nodejs_version=18.19.0 version=2.19.4 webpack_version=5.88.1 yarn_version=1.22.21
portainer | 2024/11/07 03:37PM INF github.com/portainer/portainer/api/http/server.go:357 > starting HTTPS server | bind_address=:9443
portainer | 2024/11/07 03:37PM INF github.com/portainer/portainer/api/http/server.go:341 > starting HTTP server | bind_address=:9000
traefik | 2024-11-07T15:37:47Z ERR Unable to obtain ACME certificate for domains error="cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.32.248:443: connect: connection refused" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=[""] providerName=leresolver.acme routerName=edge@docker rule=Host(``)
traefik | 2024-11-07T15:37:47Z ERR Unable to obtain ACME certificate for domains error="cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.32.248:443: connect: connection refused" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=[""] providerName=leresolver.acme routerName=frontend@docker rule=Host(``)
traefik | 2024-11-07T15:37:54Z ERR Unable to obtain ACME certificate for domains error="cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.32.248:443: connect: connection refused" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=[""] providerName=leresolver.acme routerName=frontend@docker rule=Host(``)
traefik | 2024-11-07T15:37:54Z ERR Unable to obtain ACME certificate for domains error="cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.32.248:443: connect: connection refused" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=[""] providerName=leresolver.acme routerName=edge@docker rule=Host(``)
traefik | 2024-11-07T15:37:54Z ERR Unable to obtain ACME certificate for domains error="cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.32.248:443: connect: connection refused" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=[""] providerName=leresolver.acme routerName=dashboard@docker rule="Host(``) && (PathPrefix(`/`) || PathPrefix(`/`))"
Solution
So for anyone stumbling onto this question in the future I managed to solve it, thanks to a ton of more research. What I had to do is a hard reset of the network config for the host machine:
pkill docker
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
ip link del docker0
That's it, after that I ran docker-compose and everything worked.
- Docker Compose wait for container X before starting Y
- How to use rules_oci to run an image local via Docker?
- Getting error "Get http://localhost:9443/metrics: dial tcp 127.0.0.1:9443: connect: connection refused"
- How to install pyarrow on an Alpine Docker image?
- Unable to push image to ACR from terraform - could not get auth config (the credentialhelper did not work or was not found):
- Docker returning Error 133: Failed to create temporary file with plenty of disk space
- Database's backup location that created by Azure Data Studio on Mac M1
- ubuntu wslregisterdistribution failed with error: 0x80041002
- System.InvalidOperationException: Failed to deploy distro docker-desktop to <localpath>: exit code: -1
- Installing specific version of NodeJS and NPM on Alpine docker image
- intellij idea wsl 2 can't load projet from \\WSL.LOCALHOST\[distrib]
- Docker "protocol not available" after context change
- How to configure different dockerfile for development and production
- Is it possible to convert the publish method of an App Service from Code to Docker?
- Jib create folder for temporary files and changing folder ownership
- Heroku: How to release an existing image in gitlab CI/CD?
- Access to full /proc in Docker container
- lookup registry-1.docker.io: no such host
- Nestjs doesn't handle request from nginx proxy_pass using docker-compose
- PackagesNotFoundError : The following packages are not available from current channels:
- Prisma openssl version issue
- why are alpine versions of node throwing an error that @jridgewell/gen-mapping module not found?
- Safe ways to specify postgres parameters for healthchecks in docker compose
- How to get a list of images on docker registry v2
- Docker is in volume in use, but there aren't any Docker containers
- PhpStorm inserts wrong client_host IP when trying to run debug session
- How to Properly Inject Service Account Credentials in Google Cloud Build for Cloud Run Deployment?
- Docker - COPY file from different working directory to another is not working
- starting container process caused: exec: "fastapi": executable file not found in $PATH: unknown
- docker login fails while using docker-credential-secretservice