In production we are using jetty 9.2.9.v20150224 with the following configuration:
new Server(new QueuedThreadPool(200, 5, 30000, new ArrayBlockingQueue<Runnable>(128)));
maxThreads = 200, minThreads = 5, idleTimout = 30000 ms
I tested our application and it was able to handle 200 requests/sec, but few of our clients complain that sometime even with very less load jetty does not accept any new connection and from the log that's what I found STARTED,5<=18<=200,i=4,q=128. Which as far as I understand shows there are 18 live threads in the thread pool which is way less then 200, but somehow the new connection is still rejected. Following is the snipped from the log:
2021-01-08 00:20:47,813 WARN [qtp1720926658-14444] QueuedThreadPool : QueuedThreadPool[qtp1720926658]@669341c2{STARTED,5<=18<=200,i=4,q=128}[ReservedThreadExecutor@58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@263c181
2021-01-08 00:20:47,813 WARN [qtp1720926658-14444] QueuedThreadPool : QueuedThreadPool[qtp1720926658]@669341c2{STARTED,5<=18<=200,i=3,q=128}[ReservedThreadExecutor@58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@431081c5
2021-01-08 00:20:47,813 WARN [qtp1720926658-14445] QueuedThreadPool : QueuedThreadPool[qtp1720926658]@669341c2{STARTED,5<=18<=200,i=2,q=128}[ReservedThreadExecutor@58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@3866d8cd
2021-01-08 00:20:47,813 WARN [qtp1720926658-14442] QueuedThreadPool : QueuedThreadPool[qtp1720926658]@669341c2{STARTED,5<=18<=200,i=6,q=126}[ReservedThreadExecutor@58a5954d{s=1/4,p=2}] rejected CEP:SocketChannelEndPoint@4aa81d65{/10.0.1.29:53143<->/10.0.1.28:7777,OPEN,fill=FI,flush=-,to=16442/30000}{io=1/0,kio=1,kro=1}->HttpConnection@32cdb0b1[p=HttpParser{s=START,0 of -1},g=HttpGenerator@22c521c0{s=START}]=>HttpChannelOverHttp@e02e50f{r=1,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
2021-01-08 00:20:47,829 WARN [qtp1720926658-14442] EatWhatYouKill :
java.util.concurrent.RejectedExecutionException: CEP:SocketChannelEndPoint@4aa81d65{/10.0.1.29:53143<->/10.0.1.28:7777,OPEN,fill=FI,flush=-,to=16442/30000}{io=1/0,kio=1,kro=1}->HttpConnection@32cdb0b1[p=HttpParser{s=START,0 of -1},g=HttpGenerator@22c521c0{s=START}]=>HttpChannelOverHttp@e02e50f{r=1,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:440)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.execute(EatWhatYouKill.java:370)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
STARTED,5<=18<=200,i=4,q=128 says ...
5 is your configured minimum threads<=18<= is your remaining threads available in the pool200 is your configured max threadsi=4 you have 4 idle threads in the pool (eligible for cleanup/removal)q=128 you have 128 tasks/jobs in the queue (this value is too high, it hints at your QTP configuration being insufficient for your load)Keep in mind that when you experience the rejected tasks/jobs it can be the queue or the pool that rejects them, we don't know which one in Jetty 9.2.x (this information is now available in the logs on Jetty 9.4.x)
This limited view of the QTP in the Jetty 9.2.x series does not show you even half of what's going on internally.
Example: in Jetty 9.4.x+ you see ...
All of which are exposed in JMX as well.