apache-kafkacqrsevent-driven-design

what message delivery guarantees to choose for KAFKA CQRS


need help i am implementing CQRS using i would like to know which method i should choose when it comes to message delivery guarantees which one is suitable for CQRS


Solution

  • tl;dr You want at-least-once semantics and ensure you can handle duplicates.

    So "at-most-once" is out of the question anyway, because you might lose messages.

    "Exactly-once" may sound tempting, and to be honest I'm not 100% sure how that works in Kafka, but from the page you linked you can see it's guaranteed inside the Kafka eco-system (Kafka Streams, Connect). I think it's fair to say Kafka guarantees, in some circumstances, exactly-once delivery. What you should be more concerned about is exactly-once processing in your consumer application.

    Even if a message is delivered exactly once, imagine your consumer reads it, updates the read side, but before the Kafka offset is committed, your consuming application crashes. Then, once the process is restarted, you'd still read that message again. That's to say, even with "exactly-once" delivery, you need to be able to handle duplicates. Kafka's "exactly-once" does not turn Kafka into a transactional system that takes part in distributed two-phase-commit transactions (and not that anyone would want that, anyway).

    How to handle duplicates - that is very dependent on your messages. Maybe they're idempotent anyway, maybe you need to build in some deduplication.

    But if you can handle duplicates - then "at-least-once" is just fine. So that's it, "at-least-once" plus the ability to handle duplicates will work fine for you (and this is, from the projects I've seen, in fact how people usually do it).