To set the context, we have a legacy server that services connections coming in from a large a number of clients. The connections are tcp and uses a binary protocol. Lately as the clients have increased, the server seems to be unable to handle all the connections that comes in especially if the server ever goes for a restart. In such case the server goes into some inconsistent state, with losts of connections in CLOSE_WAIT.
If the clients connection requests are staggered the server eventually takes the load. We verified this by simulating the clients in docker.
Since there is no rate limiting built into the server, I decided to try out iptables to see if I can use the rate limiting feature in iptables to stagger incoming connection requests. The connections can originate from same ip or different ip.
The rate limiting should not affect any established connections but only the newer ones(3 way handshake). Since the clients have a retry logic they would eventually come and reconnect. So do not want to rate limit by ip because all the connections even if from the same source ip should eventually connect without disturbing the established connections.
This is the iptables rules that I am using. Is this the right way?
iptables -A INPUT -i eth0 -p tcp --dport 9000 -m state --state NEW,ESTABLISHED -m recent --set -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 9000 -m state --state NEW -m recent --update --seconds 600 --hitcount 11 -j REJECT --reject-with tcp-reset
I am not familiar with iptables and is currently using ufw wrapper on Ubuntu 14.04 . The above idea was sourced from this link https://www.cyberciti.biz/tips/howto-limit-linux-syn-attacks.html.
The intention is to prevent the SYN flood but eventually slow and allow all clients to establish connection.
Just figured this out after a long a trial and error effort. The solutions is to use the iptables limit module to rate limit the connection any new incoming connections.
Ended up adding the following chain in /etc/ufw/before.rules
-A ufw-before-input -p tcp --dport 9000 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -p tcp --dport 9000 -m state --state NEW -m limit --limit 4/min --limit-burst 4 -j ACCEPT
-A ufw-before-input -p tcp --dport 9000 -j DROP
References