apache-kafkapipelineevent-sourcing

Is there a way that Service A would do the data message and publish it back to the same topic and Service B would consume it?

We are just starting to use Kafka and I am just getting to know the the different functionalities of Kafka. we have a requirement, where an event sent thru a Kafka topic needs to be consumed one service (say Service A). Service A does some data messaging to that event and publishes back to Kafka topic. And it would then need to be consumed by Service B and so on. So this can be implemented by using different topics for Service A, B and C. But I am wondering if there is more efficient way to do this, using some of Kafka's capabilities (which I am not very sure of). Is there a way that Service A would do the data message and publish it back to the same topic and Service B would consume it. So basically is there a way of kafka messages to be consumed by services in sequence? Is event sourcing a way to implement this? Thank you in advance!

I tried to have individual topics for each of the services. I am wondering if we there is a more efficient to implement this.

Solution

  • publish it back to the same topic

    Don't do this, it'll create an infinite loop... Kafka can handle hundreds of thousands of topics so just make a new one. That's how it's designed, so there's no more efficient way. Otherwise, sounds like you should use Kafka Streams or ksqlDB

    if we have to create like 4 topics for each of the microservices, is there any real advantage of using Kafka here ? I feel, we could also use any messaging point to point interface.

    1. Your apps can create their own topics. Topics do not need created ahead of time. In fact, both options I mentioned will create even more "internal" topics. Kafka has been documented as handling hundreds of thousands of topics and partitions; they're cheap.

    2. (Call me biased), but p2p messaging like HTTP / gRPC is subject to data loss with skipped / missed records. While there's a cost tradeoff of running any message broker cluster, you'll otherwise need to setup load balancers and have high availability of the receiving side of all "points" in your p2p design. Kafka is a distributed transaction log, not exclusively a message queue, such as Redis or AWS SQS, meaning you can reprocess the data for multiple purposes.