javaapache-kafkaapache-camelkafka-consumer-apikafka-partition

Create concurrent Kafka consumers in Camel-Kafka

I'm using camel-kafka version 2.14.3 . Below is the kafka URI :

<from uri="kafka:{brokerlist}?topic={topic-name}&amp;zookeeperHost={zookeeperHost}&amp;zookeeperPort={zookeeperPort}&amp;groupId={groupId-name}&amp;consumerStreams=2" />

note that I have used consumerStream=2 in URI options. But when I publish multiple messages to the topic topic-name at once (all to the same partition), the kafka consumer receives those messages sequentially. How can receives those messages parallely?

I'm looking for a solution like below :

<from uri="ibm_bean_name:queue_name?concurrentConsumers=2" />

is what I use to read concurrently from an ibm MQ

Solution

  • The partition is Kafka's unit of parallelism. Each partition will be assigned to a single consumer in a given consumer-group. You said you are producing to a single partition. That means all the messages will go to a single consumer, regardless of how many consumers you have in the group. If you want the messages to go to different consumers, you need to produce to different partitions.