apacheamazon-web-servicesamazon-elbffserver

Amazon Elastic Load Balancer is not closing the connection to the server

I have an EC2 instance with Apache as a reverse proxy and ffserver as a streaming server. There is an ELB (Classic) in front of the EC2 instance which works as an SSL termination point.

Apache configuration is rather simple:

<Location "/mp3/">
    ProxyPass http://127.0.0.1:8081/ DisableReuse=On KeepAlive=Off
    ProxyPassReverse http://127.0.0.1:8081/
    SetEnv force-proxy-request-1.0.1
    SetEnv proxy-nokeepalive 1
</Location>

ffserver is used to stream live audio over the Internet. In ffserver's settings there is a MaxBandwidth option (default 1000). This setting become a problem when the connections are not closed properly. ffserver starts to respond with 503 server too busy instead of the stream's content.

If I connect to the server directly (no ELB on the road) everything works fine. If I connect over the ELB the connection won't close when I close it on the client's side (e.g. close the browser's tab).

I use the below command to check the current connections:

watch -n 2 'netstat -napt | grep 8081'

All of the connections remain in ESTABLISHED state forever (at least for 30 minutes). The default idle timeout for ELB is 60. It means that someone is receiving the stream from ffserver (connection is active).

EDIT: It looks like changing the Classic Load Balancer to the Application Load Balancer solved the problem. I don't know how to explain this behavior. Looking forward for an answer from AWS community - AWS Forums

Solution

  • As OP shared with his edit, the issue of connections not getting closed can be resolved with a change of load balancer type. This answer focuses on why this change has such an affect?

    There seems to be an issue within Classic Load Balancer (ELB). I've found the following posts with very similar issues;

    It seems the issue stems from ELB not being able to detect the client dropping from a connection. Especially when back-end is supplying some sort of data in a periodical manner, e.g. live audio stream, heartbeat, etc.

    There doesn't seem to be a way to disable the keep-alive setting of load balancers, yet, somehow, only with ELB this trouble occurs.

    I couldn't find exact feature that is creating this behavioural difference between ELB & ALB. I think the reason is either due to;

    I think the issue resolves when Application Load Balancer (ALB) is used due to said improvements and it being much more flexible.

    Check here for more on the differences between ELB, ALB, and NLB

    ps. AWS support forums are terrible, all the good support & tips are paywalled and stored within PMs between them & their premium customers.