Multiple consumers sharing the same ConsumerGroupId
read data from multiple topics (from multiple threads if that matters). All topics have multiple partitions, AutoCommit
is set to false for all consumers. The result is all topics end up with unread messages, while consumers have no data to consume as far as Confluent client is concerned until consumers get forcibly reconnected. When the whole process restarts and consumers get reconnected they read a random number of messages again, reducing the overall unread messages count, but just for once.
What is likely the issue and how do we investigate it further?
By definition, as far as the broker knows, everything is always going to be unread since you're not committing any offsets. You will have to maintain that logic somewhere shared in your consumer instances in order to keep your consumer positions and know what has been processed vs just consumed