ActiveMQ Artemis performance degradation compared to Classic

I'm working on a migration from ActiveMQ Classic 5.15.4 to ActiveMQ Artemis 2.17.0, and I observed a performance degradation. I tested with 1 producer on a topic and different number of consumers consuming that topic. I'm measuring the time between the creation of the message and its reception by the consumer.

The tests are done on a cluster of 3 nodes all connected to each others. Each broker is embedded in a JBoss. I used a cluster of 3 nodes because that's our current production setup. I'm challenging this setup because we have few consumers and producers (less than 50 each time) and we are using message grouping but I need to do POC on a setup with only 2 nodes in active/standby mode.

The producer is targeting always the same node and the consumers are connected to the other 2 nodes randomly.

We can see that for all cases, Artemis is slightly slower than ActiveMQ Classic. I'm wondering if this is something expected.

Generally speaking, ActiveMQ Artemis is notably faster than ActiveMQ Classic due to the significant architectural differences between them. In short, ActiveMQ Artemis was designed to be completely non-blocking and performs very well at scale compared to ActiveMQ Classic.

However, in this case you are not testing the brokers at scale. You are testing one producer and a "different number" of consumers. This is certainly not the kind of production use-case that would warrant a cluster of 3 brokers. A single broker on modest or even minimal hardware would almost certainly suffice.

Even if you push the client count to around 50 I still think one live node would suffice. You definitely want to use just one live node if you're using message grouping. See the documentation for important details about clustered message grouping.

It's also important to keep in mind that you must compare "apples to apples" in terms of each broker's configuration. This is not necessarily trivial, especially when dealing with clusters. You did not share your broker configurations so I can't comment on whether or not they are functionally equivalent or at least as close to functionally equivalent as possible. There are many different configuration-specific reasons why one broker might perform better than the other in certain use-cases.

Over the last several years SoftwareMill has published benchmarks for popular message brokers for the persistent, replicated queue use-case. The last time both ActiveMQ Classic and Artemis were tested was in 2017. Here are the results. Since then SoftwareMill no longer tests ActiveMQ Classic.