Large delays in consumption of grouped messages in ActiveMQ Classic

We have a network of brokers consisting of three brokers on different servers. The network connectors are configured as follows:

<networkConnectors>
   <networkConnector uri="static:(ssl://broker2:61616,ssl://broker3:61616)" networkTTL="5"/>
</networkConnectors>

The overall setup looks like what is shown in the image below.

In practice we sometimes see large delays for the message consumption of a specific client. In that case, messages are forwarded multiple times to other brokers before finally being consumed. Until now, we have been unable to find out what might be the cause of this.

The consumers are not very busy. They are implemented using the Spring JMS library. The minimum number of consumers is 3 per broker, but this can automatically scale depending on the need.

We suspect that the reason that only a specific client is impacted, might have something to do with the fact that we use the JMSGroupId to get some ordering in how the messages are handled. However, we have no sure proof of this. And it still doesn't explain why it happens at all.

We are also considering to add the following parameters to the network connector as we think this might improve the behavior:

dynamicOnly="true"
decreaseNetworkConsumerPriority="true"
suppressDuplicateQueueSubscriptions="true"

However, it is also scary to do as we feel we do not fully understand what is happening right now and so cannot really be sure of the effect these settings will have on the behavior.

We suspect that the reason that only a specific client is impacted, might have something to do with the fact that we use the JMSGroupId to get some ordering in how the messages are handled.

Bingo.

Scaling consumer is useless if your ensure serial message processing. The best way to get rid of this behavior is to set up a dedicated JMS queue with a single consumer per group, otherwise you will experiment consumer contention when multiple following messages in the queue belong to the same group.

See JMS topology (Queue with multiple consumers) and message groups