nginx-proxy+docker swarm => load balancer not working

I am trying to use nginx-proxy with Docker Swarm mode. I have a stack for nginx-proxy and a stack for a whoami container which is replicated 3 times.

THe problem is that when I query the cluster, I always get response from the same whoami. When I hit docker service scale whoami_whoami=1 then nginx replies with 503 status code which means it keeps asking one of the removed replicas.

# proxy.yml
version: '3.8'

networks:
  proxy:
    driver: overlay
    name: proxy

services:
  nginx:
    image: jwilder/nginx-proxy:latest
    networks:
      - proxy
    environment:
      - HTTPS_METHOD=noredirect
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      # - ${CLU_ROOT}/proxy/certs:/etc/nginx/certs:ro
    ports:
      - 80:80
      # - 443:443
    deploy:
      placement:
        constraints:
          - node.role == manager

# whoami.yml
version: '3.8'

networks:
  proxy:
    external: true

services:
  whoami:
    image: jwilder/whoami:latest
    networks:
      - proxy
    environment:
      - VIRTUAL_HOST=whoami.dev.interne.eseo.fr
    deploy:
      replicas: 3

I was thinking that nginx will send requests to "whoami" and that it was the role of Swarm to balance between replicas. Am I wrong ?

On the other hand, when I publish the port 8000 of the whoami service and hit: while sleep 1; do clear; curl whoami....:8000; done then every replies come from a different replica.

Thanks in advance

The template file provided by nginx-proxy is bad. Instead of using the DNS name of the service (whoami_whoami in my case) it uses the IP of one of the containers of the service (that's why I was always getting the same container).

Once the template file is fixed nginx-proxy make use of the Swarm mode load balancer to connect to replicas.

To fix it, 2 tasks :

1. Update the stack file to mount a custom nginx.tmpl:

# proxy.yml

...

services:
  nginx:
    image: jwilder/nginx-proxy:latest
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./nginx.tmpl:/app/nginx.tmpl:ro  # <=== Add this line to your service.
    ...

2. Update the "define upstream" section in the nginx.tmpl file.

{{ define "upstream" }}
  server {{ index .Container.Labels "com.docker.swarm.service.name" }}:{{ .Address.Port }};
#   {{ if .Address }}
#       {{/* If we got the containers from swarm and this container's port is published to host, use host IP:PORT */}}
#       {{ if and .Container.Node.ID .Address.HostPort }}
#           # {{ .Container.Node.Name }}/{{ .Container.Name }}
#           server {{ .Container.Node.Address.IP }}:{{ .Address.HostPort }};
#       {{/* If there is no swarm node or the port is not published on host, use container's IP:PORT */}}
#       {{ else if .Network }}
#           # {{ .Container.Name }}
#           server {{ .Network.IP }}:{{ .Address.Port }};
#       {{ end }}
#   {{ else if .Network }}
#       # {{ .Container.Name }}
#       {{ if .Network.IP }}
#           #server {{ .Network.IP }} down;
#       {{ else }}
#           server 127.0.0.1 down;
#       {{ end }}
#   {{ end }}
{{ end }}

The original nginx.tmpl file can be downloaded from here : https://raw.githubusercontent.com/nginx-proxy/nginx-proxy/master/nginx.tmpl.