When I execute netstat -tulnp, the output is as follows:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.11:43043 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8005 0.0.0.0:* LISTEN 1/java
tcp 0 0 0.0.0.0:2021 0.0.0.0:* LISTEN 1/java
tcp 0 0 0.0.0.0:22222 0.0.0.0:* LISTEN 1/java
tcp 0 0 0.0.0.0:8719 0.0.0.0:* LISTEN 1/java
tcp 101 0 0.0.0.0:80 0.0.0.0:* LISTEN 1/java
tcp 51 0 0.0.0.0:1234 0.0.0.0:* LISTEN 1/java
tcp 0 0 0.0.0.0:20891 0.0.0.0:* LISTEN 1/java
udp 0 0 127.0.0.11:55285 0.0.0.0:* -
The value of Recv-Q caught my attention. After my investigation, I found that OOM occurred in the JVM application, and it can be found in the log that the http-nio-80-Acceptor-0 thread responsible for monitoring port 80 has exited, and the thread responsible for dispatching port 1234 requests has exited. The relevant logs are as follows:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-80-Acceptor-0"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-5"
The default configuration used by tomcat, that is, the backlog is 100, and the source code is located at tomcat/AbstractEndpoint.java at 8.5.59 · apache/tomcat · GitHub:
/**
* Allows the server developer to specify the acceptCount (backlog) that
* should be used for server sockets. By default, this value
* is 100.
*/
private int acceptCount = 100;
public void setAcceptCount(int acceptCount) { if (acceptCount > 0) this.acceptCount = acceptCount; }
public int getAcceptCount() { return acceptCount; }
The listening of port 1234 is triggered and created by HTTPServer, and the created code is HttpServer.create(new InetSocketAddress(PROMETHEUS_SERVER_PORT), 0);, backlog is corrected to 50 in ServerSocket.java, the source code is located at jdk/ServerSocket.java at jdk8-b120 · openjdk/jdk · GitHub:
public void bind(SocketAddress endpoint, int backlog) throws IOException {
if (isClosed())
throw new SocketException("Socket is closed");
if (!oldImpl && isBound())
throw new SocketException("Already bound");
if (endpoint == null)
endpoint = new InetSocketAddress(0);
if (!(endpoint instanceof InetSocketAddress))
throw new IllegalArgumentException("Unsupported address type");
InetSocketAddress epoint = (InetSocketAddress) endpoint;
if (epoint.isUnresolved())
throw new SocketException("Unresolved address");
if (backlog < 1)
backlog = 50;
try {
SecurityManager security = System.getSecurityManager();
if (security != null)
security.checkListen(epoint.getPort());
getImpl().bind(epoint.getAddress(), epoint.getPort());
getImpl().listen(backlog);
bound = true;
} catch(SecurityException e) {
bound = false;
throw e;
} catch(IOException e) {
bound = false;
throw e;
}
}
From netstat(8) - Linux manual page we know, Recv-Q indicates current syn backlog when socket is in Listening state, what confuses me is why Recv-Q is one more than the value of backlog we set?
A backlog value of N really does mean allow "N + 1" connections to queue to a listening socket. This allows one to specify "0" as the backlog and still get 1 connection.
Reference: NET: Revert incorrect accept queue backlog changes. · torvalds/linux@64a1465 · GitHub