Weblogic JMS client throws DispatcherException when trying to commit after long processing

I am working on a JMS client for WebLogic 11g (WebLogic Server 10.3.6.0). The client is based on wlthint3client.jar library. The client uses asynchronous API (listeners) to read messages from JMS queue. Client is multi-threaded and uses multiple parallel JMS consumers.

Weblogic JMS server is configured to have a JMS transaction timeout of 10 minutes, yet I have problems with commiting transactions that are over 6 minutes long. This happens only when there are many long transactions going at the same time.

Is such case commit fails with following exception:

weblogic.jms.common.JMSException: weblogic.messaging.dispatcher.DispatcherException: java.rmi.RemoteException: Could not establish a connection with -6273788781435586093S:host:[7001,7001,-1,-1,-1,-1,-1]:domain:server, java.io.IOException: Bootstrap request to JVMID -6273788781435586093S:host:[7001,7001,-1,-1,-1,-1,-1]:domain:server got an error or timed out; nested exception is:
    java.io.IOException: Bootstrap request to JVMID -6273788781435586093S:host:[7001,7001,-1,-1,-1,-1,-1]:domain:server got an error or timed out; nested exception is:
    java.rmi.ConnectException: Could not establish a connection with -6273788781435586093S:host:[7001,7001,-1,-1,-1,-1,-1]:domain:server, java.io.IOException: Bootstrap request to JVMID -6273788781435586093S:host:[7001,7001,-1,-1,-1,-1,-1]:domain:server got an error or timed out; nested exception is:
    java.io.IOException: Bootstrap request to JVMID -6273788781435586093S:host:[7001,7001,-1,-1,-1,-1,-1]:domain:server got an error or timed out
    at weblogic.jms.dispatcher.DispatcherAdapter.convertToJMSExceptionAndThrow(DispatcherAdapter.java:116)
    at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncNoTran(DispatcherAdapter.java:61)
    at weblogic.jms.client.JMSSession.commit(JMSSession.java:1224)
    at weblogic.jms.client.JMSSession.commit(JMSSession.java:1198)
    at weblogic.jms.client.WLSessionImpl.commit(WLSessionImpl.java:108)

Some other observations:

It seems my client only can handle 5 messages at a time. I have more consumers, let say 10, I have more available messages in the queue, but still only 5 are processed in parallel.
If I configure the client to use < 5 consumers then the problem does not occur.
If there are < 5 parallel long transactions - the problem does not occur

Solution

So the solution of the problem is to set a property on JVM running the client application -Dweblogic.ThreadPoolSize=N+1, where N is number of asynchronous listeners used by the client.

If you set it equal to N, and N messages are received and stuck for more than 4 minutes, then your commit will fail with aforementioned exception.

Why? I guess when you have all the listener threads busy processing incoming messages, there is nobody there to talk to the server and keep the connection alive. This is confirmed by Oracle documentation on T3 connection heartbeats:

Any two Java programs with a valid T3 connection — such as two server instances, or a server instance and a Java client — use periodic point-to-point "heartbeats " to announce and determine continued availability [...]

and

The frequency with which a server instance issues heartbeats is determined by the heartbeat interval, which by default is 60 seconds.

The number of missed heartbeats from a peer that a server instance waits before deciding the peer is unavailable is determined by the heartbeat period, which by default, is 4. Hence, each server instance waits up to 240 seconds, or 4 minutes, with no messages—either heartbeats or other communication—from a peer before deciding that the peer is unreachable.

and in another place:

WebLogic client thread pools are configured differently than WebLogic server thread-pools, and are not self tuning. WebLogic clients have a specific thread pool that is used for handling incoming requests from the server, such as JMS MessageListener invocations. This pool can be configured via the command-line property:

-Dweblogic.ThreadPoolSize=n

enter link description here