apache-kafkastrimziapache-kafka-mirrormaker

Kafka MirrorMaker2 automated consumer offset sync

I am using MirrorMaker2 for DR.

Kafka 2.7 should support automated consumer offset sync

Here is the yaml file I am using (I use strimzi for creating it)

Code

All source cluster topics are replicated in destination cluster. Also ...checkpoint.internal topic is created in destination cluster that contains all source cluster offsets synced, BUT I don't see these offsets being translated into destination cluster _consumer_offsets topic which means when I will start consumer (same consumer group) in destination cluster it will start reading messages from the beginning.

My expectation is that after allowing automated consumer offsets sync all consumer offsets from source clusters translated and stored in _consumer_offsets topic in the destination cluster.

Can someone please clarify if my expectation is correct and if not how it should work.

Solution

  • The sync.group.offsets.enabled setting is for MirrorCheckpointConnector.

    I'm not entirely sure how Strimzi runs MirrorMaker 2 but I think you need to set it like:

    checkpointConnector:
  config:
    checkpoints.topic.replication.factor: 1
    sync.group.offsets.enabled: "true"