eventsapache-kafkakafka-consumer-api

How a consumer able to get all messages from a particular partition of a topic?


I was going the Kafka tutorials and lead to one confusion. A topic is distributed in multiple partitions. A producer message goes only into one partition not all partition of a topic. When a consumer subscribe topic it is assigned to a particular partition and it reads from that. If above is true then how consumer can read all messages send to a topic when a partition does not get all the messages?

Please rectify if my assumption is wrong.

Thanks


Solution

  • Consumers can be assigned multiple partitions of a topic. Especially, if there is only one consumer in the consumer group.

    Only when there's multiple group members, will each instance process a subset of the topic. Together, they consume all messages, of all partitions.

    Documentation / tutorials often intermix "consumer" with the overall reading process (all instances are reading the entire topic), and an individual one (a consumer joins a group, and is assigned partitions)