haproxy

Haproxy agent-check DRAIN(agent) status still accepts new connections

I am trying to perform a load balancing for my Node.js application which uses websockets. I need haproxy to stop load balance new connections on a server, which has reached its maximum number of connections, keeping the existing ones intact in the same time.

I do this by performing an agent-check for each of my servers. If a server cannot accept new connections it responds with "drain" to an agent-check. If a server is able to respond to a new connection, it responds with "ready" to an agent-check.

Here is my haproxy.cfg configuration file:

global
 daemon
 maxconn 240000
 log /dev/log local0 debug
 log /dev/log local1 notice
 tune.ssl.default-dh-param 2048
defaults
 mode http
 log global
 option httplog
 option dontlognull
 option dontlog-normal
 option http-server-close
 option redispatch
 timeout connect 20000ms
 timeout http-request  1m
 timeout client 2100000ms
 timeout server 2100000ms
 timeout queue  30s
 timeout check  5s
 timeout http-keep-alive 180s
 timeout tunnel 3600s
 timeout tarpit          60s

frontend stats
    bind *:8084
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if TRUE

frontend test
    mode http
    bind *:5000

    default_backend ws


backend ws
    mode http
    fullconn 100000
    balance roundrobin
    cookie SERVERID insert indirect nocache

    server 1 backend1:9999 check agent-check agent-port 8080 cookie 1 inter 500 fall 1 rise 2

And here is how I respond to haproxy agent-check in my Node.js app:

const healthCheckServer = net.createServer((c) => {
    let data = '';
    if (currentConn < MAX_CONN) {
        data += 'ready';
    } else {
        data += 'drain';
    }
    c.write(data + '\r\n');
    c.destroy();
});

healthCheckServer.listen(8080, '0.0.0.0');

When number of connections to my application is reaching its maximum, haproxy correctly changes server status to DRAIN (agent) (I can observe this in haproxy web dashboard). The problem is, that new connections are still accepted by the application.

I am new to haproxy, so can someone point me to where I am wrong?

Solution

  • Found out that when server is drained by agent (status is set to DRAIN (agent)) and if server is the only one existing in the backend it will still accept new connections.

    When there are multiple servers present and each server is drained the behavior is just like expected: haproxy returns HTTP 503.

    UPDATE: Turned out that I was looking in the wrong direction all the time.

    First I had to mark my backend which processes WebSocket connections as a non-http (remove mode http line). My guess is haproxy incorrectly counts current sessions for http backend when using WebSockets. Removing mode http solved my problem.

    Second, returning maxconn:<conn> in agent-check looks like much simple and more idiomatic way to limit the number of concurrent connections.

    Sources:

    1. https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-agent-check
    2. https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/

    I hope this will help someone.