nettynlbproxy-protocolnetty4aws-nlb

AWS's Proxy Protocol v2 Breaking Application Due to Absence of PSH Flag


I have a network application built using Netty. The application is behind Amazon network load balancer.

I now want to be able to retrieve the original client IP address, so I turned on the Proxy Protocol v2 setting on the network load balancer.

Unfortunately doing this breaks the application.

The application can be interacted with, via the terminal using something like nc or telnet.

On connecting to the application, the user will normally receive a welcome message and then a prompt to type in the query to interact with the application.

When the Proxy Protocol v2 is turned on, on connection, the welcome message is no longer written out to the client, instead the client sees something like this:

telnet -4 app.host 43
Trying <IP ADDRESS>...
Connected to some_network_lb.elb.eu-west-1.amazonaws.com.
Escape character is '^]'.

capturing the network packet I noticed a difference between when the Proxy protocol v2 is switched off (Application works fine) and when it is switched on (Application does not work fine)

When the Proxy protocol v2 is switched off, the packet interaction can be summarized as follows:

No.     Time           Source                Destination           Protocol Length Info                                                            port
      1 0.000000       SOURCE-IP        DEST-IP        TCP      78     57059 → 43 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=1158853985 TSecr=0 SACK_PERM=1 43

No.     Time           Source                Destination           Protocol Length Info                                                            port
      2 0.034526       DEST-IP         SOURCE-IP        TCP      74     43 → 57059 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=1420 SACK_PERM=1 TSval=3185991482 TSecr=1158853985 WS=128 57059

No.     Time           Source                Destination           Protocol Length Info                                                            port
      3 0.034556       SOURCE-IP        DEST-IP          TCP      66     57059 → 43 [ACK] Seq=1 Ack=1 Win=132352 Len=0 TSval=1158854019 TSecr=3185991482 43

No.     Time           Source                Destination           Protocol Length Info                                                            port
      4 0.067278       DEST-IP         SOURCE-IP        TCP      264    43 → 57059 [PSH, ACK] Seq=1 Ack=1 Win=26880 Len=198 TSval=3185991515 TSecr=1158854019 [TCP segment of a reassembled PDU] 57059

No.     Time           Source                Destination           Protocol Length Info                                                            port
      5 0.067301       SOURCE-IP        DEST-IP          TCP      66     57059 → 43 [ACK] Seq=1 Ack=199 Win=132096 Len=0 TSval=1158854051 TSecr=3185991515 43

And when the Proxy protocol v2 is switched on, the packet interaction can be summarized as follows:

No.     Time           Source                Destination           Protocol Length Info                                                            port
      1 0.000000       SOURCE-IP        DEST-IP        TCP      78     55871 → 43 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=1158086997 TSecr=0 SACK_PERM=1 43

No.     Time           Source                Destination           Protocol Length Info                                                            port
      2 0.040204       DEST-IP        SOURCE-IP        TCP      74     43 → 55871 [SYN, ACK] Seq=0 Ack=1 Win=26847 Len=0 MSS=1420 SACK_PERM=1 TSval=3185217396 TSecr=1158086997 WS=128 55871

No.     Time           Source                Destination           Protocol Length Info                                                            port
      3 0.040227       SOURCE-IP        DEST-IP        TCP      66     55871 → 43 [ACK] Seq=1 Ack=1 Win=132352 Len=0 TSval=1158087035 TSecr=3185217396 43

As can be seen above, when the Proxy Protocol v2 is turned on, the packet exchange stops at the third exchange, and the server never sends the 4th package exchange, which contains the [PSH, ACK] when the Proxy Protocol v2 is turned off.

Any idea why this is happening? Why is it that, the packet with the [PSH, ACK] flag is never sent when Proxy Protocol v2 is turned on? And any tips on how to fix this?


Solution

  • We had to contact AWS support to help with setting the target group attribute proxy_protocol_v2.client_to_server.header_placement to on_first_ack on the NLB target group.

    By default trying to set the attribute does not work. But once AWS has enabled it, the command to update the attribute will look like this:

    “aws elbv2 modify-target-group-attributes --target-group-arn arn:aws:elasticloadbalancing:us-east-2*************************** --attributes ‘{“Value”: “on_first_ack”, “Key”: “proxy_protocol_v2.client_to_server.header_placement”}’ ”
    

    For us, this was due to the nature of our application, where no data is sent from the client after the connection is made, and this does not work out of the box because headers are pushed lazily.

    You can read this link for more info on this behaviour