Message are not distributed to virtual topic consumers

We are using ActiveMQ Artemis 2.19.1 with a primary/backup HA pair of brokers using shared storage via NFSv4.

We have a virtual topic setup defined with

virtualTopicConsumerWildcards=clustered.*.>;2;selectorAware=true

We have 1 topic called topic.update.V1.

And we have 5 consumers on this topic.

• 2 are direct topic consumers
• 3 are virtual topic consumers

When a message is produced, we can see this in the logs

***** durable queues topic.update.V1:
- queueID=2192 address:topic.update.V1 name:clustered.aaa.topic.update.V1 filter:null
- queueID=269702668 address:topic.update.V1 name:clustered.bbb.topic.update.V1 filter:null
- queueID=236513962 address:topic.update.V1 name:clustered.ccc.topic.update.V1 filter:null
***** non durable for topic.update.V1:
- queueID=32495923489 address:topic.update.V1 name:1b8afd98-1466-423d-92c1-ea15fb1c9190 filter:null
- queueID=32495922963 address:topic.update.V1 name:aabfabad-e741-4d2f-a3e5-15a58e3529b0 filter:null

Everything works fine for few hours and then the virtual consumers stop receiving the messages, only the direct consumers continue to receive the messages.

We observed in the logs that the issue correlates with the appearance of the pattern PagedReferenceImpl

We also observe that the number of messages on those queues (AddressControl.getNumberOfMessages()) is incorrect. It reports ~500,000 messages on each virtual topics that seems to be not there (storage usage does not reflect this 500,000 messages + before the issue start, new message are received correctly which should not be the case if ~500,000 messages were pending).

When we restart the cluster, the message distribution issue disappear for few hours but the ~500,000 are still reported.

Any idea what could be wrong?

I believe this is likely related to ARTEMIS-4241 given that virtual topic support is implemented using fully-qualified-queue-names and your use-case involves paging.

I strongly recommend you upgrade to the latest release which is currently 2.40.0. Version 2.19.1 was released in June 2022 almost 3 years ago at this point. Between 2.19.1 and 2.40.0 there have been over 2,300 commits and hundreds of Jiras resolved aside from ARTEMIS-4241.