goapache-kafkakafka-consumer-apisarama

how to handle consumer recovery by offset with shopify sarama

I've read that kafka provides a consumer client library that allows recovery by saving the last offset read in zookeeper (not 100% sure about where it's stored).

Is it possible to do the same with Sarama consumers?

Let's say that I'm reading until offset 550, my consumer crashes for 5 min, we are now at offset 700 but I want to resume consuming from offset 550.

Is that possible without having to save the state by myself? I would assume it does but I don't understand how.

I've found sarama.OffsetNewest/Oldestbut that's not what I'm looking for...

Solution

  • Kafka consumers used to store offsets in Zookeeper but now they store them directly in Kafka. See the Consumer section in the Kafka docs.

    Sarama handles that very well and Sarama consumers will commit (store) offsets in Kafka by default.

    Have a look at the Sarama Consumer example. Initially this example starts at the end of the topic but upon restarting, it will restart from its last position.