ISSUE
I can't see the original remote-ip coming from outside, when I use SSL-Termination in HAProxy and force HTTPS. If I do not force HTTP to HTTPS redirect, I can see from nginx-logs that send-proxy is in fact working with HTTP. I can see the same thing from HAProxy-logs too.
THEORY
If I remove all cookies of the site, switch redirect off and open the HTTP version, HAProxy will show the remote-ip correctly. After switching redirect to HTTPS and opening the same HTTP-site, logs will first show a connection from remote-ip to backend:
176.93.182.162:49300 [14/Feb/2020:23:40:30.254] 80_443_frontend 80_443_frontend/(NOSRV)
and then a connection from localhost (127.0.0.1):
127.0.0.1:35304 [14/Feb/2020:23:40:30.776] 80_443_frontend~ 80_443_backend/git 0/0/0/1/1 200
CONCLUSION
My thoughts are, that HAProxy will direct internal traffic from 443 back to port 80 towards the backend, and at the same time switch the header IP to it's own IP. Hopefully someone will have an idea.
HAProxy is inside it's own lxd container, as is NGINX!! If this changes something?
This is the configuration in HAProxy:
frontend 80_443_frontend
bind *:80
bind *:443 ssl crt /etc/ssl/git.domain.org/git.domain.org.pem alpn h2,http/1.1
redirect scheme https code 301 if !{ ssl_fc }
timeout client 1m
option forwardfor
option http-server-close
acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt_backend if letsencrypt-acl
acl git_host hdr(host) -i git.domain.org
use_backend 80_443_backend if git_host
backend letsencrypt_backend
server letsencrypt 127.0.0.1:8888
backend 80_443_backend
http-request set-header X-Client-IP %[src]
server git git.lxd:80 send-proxy
Well, f*ck me sideways!
One needs to remember what ports are forwarded to containers!!
This is what was missing:
sudo iptables -t nat -I PREROUTING -i eth0 -p TCP -d PUBLIC_IP/32 --dport 443 -j DNAT --to-destination 10.4.21.53:443