I'm a bit confused about how Apache Flink works with commit offset when Kafka is the datasource.
I have my consumer group configured and consuming messages normally, and committing it (because I enabled checkpoint on flink). But when I restart my deploy on k8s all message are consumed again even if my LAG is zero on consumer. Seems like flink is just ignoring the consumer commit offset.
I read a bit about that and seems like a need to store state somewhere but I don't understand why I need it if I have my offset commited.
Flink does its own offset management so that it can guarantee effectively exactly-once semantics. The Flink/Kafka connector commits the offsets back to Kafka as it completes each checkpoint -- but it does this as a convenience for the user. Flink does not rely on the offsets stored in the broker.
Exactly-once with Flink and Kafka requires resuming from the offsets saved in a checkpoint or savepoint. Otherwise Flink will start from whatever offsets you have configured the consumer to use.
The Kafka connectors docs are here.
For more about how Flink does exactly-once with Kafka, see https://www.youtube.com/watch?v=YEsP9zW1h10