I made some changes on Load Balancer to capture the real client IP as per edge stack docs and digitalocean annotations docs. I made the following changes in Load Balancer:
In Service:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "false"
service.beta.kubernetes.io/do-loadbalancer-protocol: http
service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
In Module:
apiVersion: getambassador.io/v3alpha1
kind: Module
metadata:
name: ambassador
namespace: ambassador
spec:
config:
xff_num_trusted_hops: 1
use_remote_address: false
After these changes we are able to detect client IP "xxx.xxx.xxx.xxx" on HTTP requests as shown in logs:
ACCESS [2022-09-13T08:43:06.434Z] "GET /v1/test/exp1 HTTP/1.1" 301 - 0 0 135 - "xxx.xxx.xxx.xxx" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" "e0c33ab9-9b8b-43a1-927b-da6c324b16b2" "xyz.example.com" "-"
But on HTTPS we are getting client IP as "-":
ACCESS [2022-09-13T08:43:18.619Z] "GET /v1/test/exp1 HTTP/1.1" 200 - 0 2 102 63 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36" "de9b20c1-53bb-447b-b336-45cd532c54c2" "xyz.example.com" "xx.xxx.xxx.xxx:80"
You are facing an issue as you are using passthrough on HTTPS but as per digital ocean docs for passthrough
SSL passthrough distributes the decryption load across the backend servers, but every server must have the certificate information. You also can’t add or modify HTTP headers, so you may lose the client’s IP address, port, and other information contained in the X-forwarded-* headers.
As suggested by DO docs we need to use a certificate so that we cannot lose HTTP headers info which includes X-Forwarded-For as well so the following steps are required to resolve this issue:
First, create a certificate using this DO API: Create Certificate.
In response you will get certificate id something like uuid("892071a0-bb95-49bc-8021-3afd67a210bf") save it for later use.
Made the following changes in ambassador service:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "false"
service.beta.kubernetes.io/do-loadbalancer-protocol: http
service.beta.kubernetes.io/do-loadbalancer-certificate-id: "<HERE COMES CERTIFICATE ID>"
service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
Do helm upgrade or kubectl apply to reflect changes.
That's it, enjoy.....