apache-kafkakafka-consumer-apiautocommitlibrdkafka

KafkaConsumer CPP API assign() with auto commit


I have a CPP Kafka consumer which uses assign to specify the partitions. Since I assign the partitions using assign() and not use the subscribe() which I am fine with. Because of this, my the re-balancing doesn't take place which also I am fine with.

Question 1: I want to understand how autocommit works here. Lets say if there are 2 consumers both of which have the same groupId. Both of them will get all the updates but could someone help me understand how commit will happen here ? If there is only one consumer, the commit happens using the consumer group id. But how does it work with 2 consumers. I don't see any commit failures as well in these cases.

Question 2: How does rd_kafka_offsets_store work when I assign partition. Do they go along well or should I make use of subscribe in these cases ?


Solution

    1. Two non-subscribing consumers with the same group.id will commit offsets for their assigned partitions without correlation or conflict resolution, if they're assigned the same partitions they will overwrite eachothers's commits. Either use unique group.id's or subscribe to the topics.
    2. rd_kafka_offsets_store() works the same way with assign() or subscribe(), namely by storing (in memory) the offset to commit on the next auto or manual commit.