I have a producer that send messages to kafka topic. My topic has 10 partitions in same consumer group and I am looking for round robin distribution of messages. I have specified RoundRobinPartitioner in producer config and even tried setting batch.size to 0 assuming that will not batch any messages and will do round robin. However, I always see only 5 partitions get messages all the time. I am not specifying any key while sending message to topic. Any idea what am I missing to get the desired behavior. I am okay to enable batching as long as these batches are distributed to all partitions. Any help here would be highly appreciated.
Producer config below.
RoundRobinPartition
EnableIdempotence,true
BatchSize, 0
no explict Key added
I have produced a few thousands of messages and couldn't really observe the distribution. Can I achieve round robin with a few thousands of messages..?
I am using kafka-clients:3.1.2.
You shouldn't set the partitioner to the old RoundRobinPartitioner as setting it manually produces the problem you described. See also this question: Kafka RoundRobin partitioner not distributing messages to all the partitions.
As in earlier Kafka client versions there was a bug which leads to uneven distribution of UniformStickyPartitioner which got the new default partitioner you should upgrade the depedency to version 3.3.0 or newer. See the following bug report for further information: https://issues.apache.org/jira/browse/KAFKA-10888
Try to use kafka client version 3.3.0 or newer.