How to manage data consistency with Chronicle queue

We have been building a bunch of micro services using Apache Kafka as a messaging bus to communicate between them. Some micro services also interact with a database or a JMS message broker (IBMM MQ) or both. We had lots of problems caused by the fact that Kafka does not guaranty exactly once resulting in lots of efforts put in place to prevent duplicates happening. In addition to this the performance we achieved is far for what is being advertised.

Because low latency is one of our main requirements as a Prof Of Concept we replaced Kafka topics with Chronicle Queues and gave it a go. Depending on the batch sizes and how hard we smashed our systems in terms of TPS (transactions per second) it came as a nice surprise that Chronicle version was anywhere between six times to two hundreds times faster than the Kafka version with an average of at least ten times faster.

Anyway low latency is only one of the requirements. Data consistency is paramount. Under no circumstances we would like to duplicate a message or loose one. Going through Chronicle Queue documentation as well as searching the web did not take me anywhere about how to use Chronicle Queue in a transactional way. Considering integration with some other resources (JMS + DB) some XA transaction like would be preferred.

So my question is what are the common patterns applied when using Chronicle Queue to assure data consistency? A link to some good article or some suggestions or a bit of code example to put me on the right direction would would suffice.

Thank you in advance.

Solution

If you need to handle the failure of a host you will need our replication service which is part of Chronicle Queue Enterprise.

However, if you want to handle the failure of a process, you can use Named Tailers. This allows you to control and continue a tailer from the index it was up to.

When you open a DocumentContext you can control whether it should continue or not.

e.g.

try (ExceptTailer tailer = queue.createTailer("name")) {

// in a loop
try (DocumentContext dc = tailer.readingDocument()) {
  if (!dc.isPresent()) {
     // do something else
  }
  // process message
} catch(Throwable t) {
  dc.rollbackOnClose();
  throw t;
}

In this example, the index will only move on if it completes normally. If it doesn't complete normally, you won't see that message again on this tailer (event on restart) unless you explicitly change the index.

You can also monitor the index in another process this way by opening the same tailer.

Here is a benchmark I did comparing Kafka to Chronicle Queue https://chronicle.software/benchmarking-kafka-vs-chronicle-for-microservices-which-is-750-times-faster/