I am wondering if there is a way to do elastic load balancing. I have read about HAProxy but it seems I need to bring down HAProxy to reconfigure it to work with more or less machines.
To make the picture more clear: I have a cluster of web backends (lets say apache + mod_rails). I can monitor the usage of the backends and bring up another machine with the same content very quickly (on the order of seconds) if traffic gets very high. However, I don't know how to make HAProxy use the additional backends without restarting it (hurting availability). Is there a way using HAProxy or some other load balancer to do this?
I was thinking there might be a way to have two load balancers for redundancy. Then I could bring down one, update its configuration, bring it back up, and then take down the other. But I don't have a good idea about how to do this.
If you need to add new servers, you have to restart it, though it's almost undetectable when you start the new process with "-sf $oldpid", as both the new and old process work in parallel.
If you need to temporarily disable a server, you have several options :
1) (the preferred one) : enable "option http-disable-on-404" and manipulate your server's check response to return 404. This will disable new connections but will still allow existing users to finish their session. Then you arrange to return 500 and you can stop your process. The advantage of this method is that you never have to touch the LB, everything is controlled from the server you're operating on. This is how most of sensible infrastructures do it.
2) the easy one : using socat, connect to the stats socket and disable the server you intend to work on :
printf "disable server 1\n" | socat stdio unix-connect:/var/state/haproxy.stat
then enable it once you're finished :
printf "enable server 1\n" | socat stdio unix-connect:/var/state/haproxy.stat
As long as you're not modifying the config, there's no reason to restart, even if it remains undetected.