apache-kafkaapache-kafka-mirrormaker

MM2 offset synchronization when consumer group not created on target due to non activity


I have secondary cluster, that till disaster has no consumers. while DR this cluster need be consumed by same consumer groups as on Primary and it requried to consume with same offset

if to set sync.group.offsets.enabled: true will offset infromation of specific sconsumer group be synced, even if no consumer group created yet on secondary cluster?

Kafka Improvement Proposal (KIP) 382: This KIP introduces MirrorMaker 2 and provides an overview of its features. It mentions that MirrorMaker 2 can sync consumer group offsets along with the replicated messages

Maybe someone had tried it practically?


Solution

  • From Kafka 2.7.0 and above, the answer is yes: if sync.group.offsets.enabled: true, MM2 will translate group offsets automatically from the MM2 checkpoints topic into the "__consumer_offsets" topic on the destination cluster.

    This an improvement that allow consumers and stream applications migrate from source to target cluster transparently, so that when the consumer and stream applications switch to the target cluster, it will resume to consume from where it left off at source cluster.