Ingress Nginx NLB get Client Real IP (Broken header: "" while reading PROXY protocol)

I've ingress nginx controller exposed via private NLB (Network Load Balancer). I want to enable host whitelisting on ingress Nginx.
My use case is to allow request from VPC1 to VPC2 and only request coming from VPC1 should be allowed to go through this private nginx. For this I've used below annotation
nginx.ingress.kubernetes.io/whitelist-source-range
The problem I got from this is that ingress-nginx was not receiving client real IP. After doing some research I found out that I've to enable proxy protocol on NLB. To do this I added following annotations and configurations.

         annotations:
              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
              service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
              service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
              service.beta.kubernetes.io/aws-load-balancer-type: nlb
              service.beta.kubernetes.io/aws-load-balancer-internal: "true"
              service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: proxy_protocol_v2.enabled=true
          metrics:
            enabled: true
          config:
            use-proxy-protocol: "true"
            real-ip-header: "proxy_protocol"

To be precise I've added only this part

       config:
            use-proxy-protocol: "true"
            real-ip-header: "proxy_protocol"

       service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: proxy_protocol_v2.enabled=true
    
I've also tried this annotation with same config
       service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

The error I'm receiving is

broken header: "" while reading PROXY protocol, client: xx.xx.xx.xx

I'm not able to figure out what I'm doing wrong. Any help is greatly appreciated.

Update 1:
I checked on aws console proxy protocol was not enabled by this annotation. When I manually enabled it everything worked. But I'm not understanding why this is not working, is it related to the version of ingress nginx I'm using ?

In order to enable proxy protocol and make this annotation work
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: proxy_protocol_v2.enabled=true
We have to use aws load balancer controller. With this annotation service.beta.kubernetes.io/aws-load-balancer-type: nlb kubernetes uses its own load balancer controller which do not support this annotation.
To use aws load balancer controller we need to add these annotations.

service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance/ip (on the basis of your use case).

For more details please refer this documentation.
If you want to dive deeper and check kubernetes codebase please follow this link.