dockercontainersreverse-proxytraefikportforwarding

Traefik 3 returns "404 page not found" on mailpit 8025 port - cannot reverse proxy to specific container port

The problem is rather general but I have an example with mailpit: I cannot reverse proxy to a docker container app which uses a specific port.

I'm using Traefik and I set up these files:

Traefik/docker-compose.yaml:

services:

  traefik:
    image: traefik:latest
    container_name: traefik
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./htpasswd:/htpasswd
      - ./traefik.yaml:/etc/traefik/traefik.yaml
      - /var/run/docker.sock:/var/run/docker.sock
    labels:
      - "traefik.http.routers.dashboard.rule=Host(`localhost`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.usersfile=/htpasswd"
    networks:
      - traefik

  whoami:
    image: traefik/whoami
    labels:
      - "traefik.http.routers.whoami.rule=Host(`localhost`) && PathPrefix(`/whoami`)"
      - "traefik.http.routers.whoami.tls=true"
    networks:
      - traefik

networks:
  traefik:
    external: true

Having this static configuration Traefik/traefik.yaml:

api:
  dashboard: true

entrypoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

log:
  level: TRACE

metrics:
  addInternals: true

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"

This works and I can run

$ curl -sk https://localhost/whoami | head 
Hostname: ea1b0e62d773
IP: 127.0.0.1
IP: ::1
IP: 172.19.0.3
RemoteAddr: 172.19.0.2:37404
GET /whoami HTTP/1.1
Host: localhost
User-Agent: curl/8.9.1
Accept: */*
Accept-Encoding: gzip

I can also access the Traefik dashboard at https://localhost/dashboard/.

However, if I add another service like e.g. mailpit:

Mailpit/docker-compose.yml:

services:
  mailpit:
    image: axllent/mailpit
    container_name: mailpit
    restart: unless-stopped
    volumes:
      - mailpit-data:/data
    ports:
      - 8025:8025
      - 1025:1025
    environment:
      MP_MAX_MESSAGES: 5000
      MP_DATABASE: /data/mailpit.db
      MP_SMTP_AUTH_ACCEPT_ANY: 1
      MP_SMTP_AUTH_ALLOW_INSECURE: 1
    labels:
      - "traefik.http.routers.mailpit.rule=Host(`localhost`) && PathPrefix(`/mailpit`)"
      - "traefik.http.routers.mailpit.tls=true"
      - "traefik.http.routers.mailpit.service=mailpit"
      - "traefik.http.services.mailpit.loadbalancer.server.port=8025"
    networks:
      - traefik

volumes:
  mailpit-data:

networks:
  traefik:
    external: true

Calling the interface via traefik results in 404 page not found.

$ curl -sk https://localhost/mailpit | head
404 page not found

Yet, the service is up and running and I can access mailpit dashboard right at http://localhost:8025, but not via traefik.

But the traefik dashboard lists the mailpit service correctly at url: http://172.19.0.4:8025 and accessing it within the traefik server itself works fine.

$ docker exec -it traefik wget -O - http://172.19.0.4:8025 | head
Connecting to 172.19.0.4:8025 (172.19.0.4:8025)
writing to stdout
<!DOCTYPE html>
<html lang="en" class="h-100">

<head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width,initial-scale=1.0">
        <meta name="referrer" content="no-referrer">
        <meta name="robots" content="noindex, nofollow, noarchive">

Yet, this output is not reported back.

What is wrong?

(Rewrote section, since Stack Overflow thinks this is spam)

I played around with different setting to no avail.

** Edit **

Traefik seems to forward the path to the service as-is as pointed by @larsks here. Stripping the path could help, but this causes another series of problems.

Therefore, the only easy solution is, adjusting the Host of the Traefik labels without any Path matchers, like this

services:
  mailpit:
    image: axllent/mailpit
    restart: unless-stopped
    volumes:
      - mailpit-data:/data
    ports:
      - 8025:8025
      - 1025:1025
    environment:
      MP_MAX_MESSAGES: 5000
      MP_DATABASE: /data/mailpit.db
      MP_SMTP_AUTH_ACCEPT_ANY: 1
      MP_SMTP_AUTH_ALLOW_INSECURE: 1
    labels:
      - "traefik.http.routers.mailpit.rule=Host(`mailpit.localhost`)"
      - "traefik.http.routers.mailpit.tls=true"
      - "traefik.http.routers.mailpit.service=mailpit"
      - "traefik.http.services.mailpit.loadbalancer.server.port=8025"
    networks:
      - traefik

volumes:
  mailpit-data:

networks:
  traefik:
    external: true

Having a dedicated host ("mailpit.localhost") for the service.

Solution

  • Traefik is working as expected. The 404 error is actually coming from the mailpit container. You have configured traefik like this:

        labels:
      - "traefik.http.routers.mailpit.rule=Host(`localhost`) && PathPrefix(`/mailpit`)"
      - "traefik.http.routers.mailpit.tls=true"
      - "traefik.http.routers.mailpit.service=mailpit"
      - "traefik.http.services.mailpit.loadbalancer.server.port=8025"

    This means when you make a request to https://localhost/mailpit, traefik forwards this request to http://mailpit:8025/mailpit. Note that the URL preserves the original request path. If you attempt to access this directly...

    $ curl http://localhost:8025/mailpit
404 page not found

    ...you will get the same error. Mailpit doesn't know it's sitting behind a prefix, but fortunately it has a --webroot option for exactly this situation. You need to include that in your configuration:

    services:
  mailpit:
    image: docker.io/axllent/mailpit
    restart: unless-stopped
    volumes:
      - mailpit-data:/data
    ports:
      - 8025:8025
      - 1025:1025
    environment:
      MP_MAX_MESSAGES: 5000
      MP_DATABASE: /data/mailpit.db
      MP_SMTP_AUTH_ACCEPT_ANY: 1
      MP_SMTP_AUTH_ALLOW_INSECURE: 1
    command:
      - --webroot=mailpit
    labels:
      - "traefik.http.routers.mailpit.rule=Host(`localhost`) && PathPrefix(`/mailpit`)"
      - "traefik.http.routers.mailpit.tls=true"
      - "traefik.http.routers.mailpit.service=mailpit"
      - "traefik.http.services.mailpit.loadbalancer.server.port=8025"
    networks:
      - traefik

volumes:
  mailpit-data:

networks:
  traefik:
    external: true

    This would solve the problem, except for one last thing: the mailpit image includes a healthcheck that periodically runs /mailpit --readyz, and it looks like that command will never return successfully when using the --webroot option on the service (because the healthcheck doesn't know about the --webroot configuration). We need to build a custom image to resolve that problem. That's pretty simple; we create a local Dockerfile like this:

    FROM docker.io/axllent/mailpit

HEALTHCHECK --interval=10s CMD ["/mailpit", "--webroot", "mailpit", "readyz"]

    And modify compose.yaml to build the image for us:

    services:
  mailpit:
    build:
      context: .
    restart: unless-stopped
    volumes:
      - mailpit-data:/data
    ports:
      - 8025:8025
      - 1025:1025
    environment:
      MP_MAX_MESSAGES: 5000
      MP_DATABASE: /data/mailpit.db
      MP_SMTP_AUTH_ACCEPT_ANY: 1
      MP_SMTP_AUTH_ALLOW_INSECURE: 1
    command:
      - --webroot=mailpit
    labels:
      - "traefik.http.routers.mailpit.rule=Host(`localhost`) && PathPrefix(`/mailpit`)"
      - "traefik.http.routers.mailpit.tls=true"
      - "traefik.http.routers.mailpit.service=mailpit"
      - "traefik.http.services.mailpit.loadbalancer.server.port=8025"
    networks:
      - traefik

volumes:
  mailpit-data:

networks:
  traefik:
    external: true

    With these two changes, access to https://localhost/mailpit resolves to the mailpit web ui. Note that this will only work after the container status is "healthy"; traefik will not direct traefik to unhealthy services.

    I have updated the example repository with all of these changes.