I have MirrorMaker2 (MM2) up and running for a migration. I need the replication with source alias prefix. For topics it works fine. The topics in the target Kafka are replicated with the source alias. The problem is with the consumer groups. They are not prefixed.
In the MM2 internal topic "mirrormaker2-cluster-offsets" I can see the proper offset but the group names are not prefixed. The group names in the topic "__consumer_offsets" are then also not prefixed. Because of this when I move the consumer groups from source to target (consumer groups in the target has correct prefix) they starts consuming from the beginning instead of the committed offset in "__consumer_offsets" (since the group information there was not prefixed). I have attached my MM2 yaml. Is this any mistake in the yaml thats causing consumer groups to not have prefix?
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
name: kafka-mirror
spec:
version: 3.9.0
replicas: 4
connectCluster: "cluster"
clusters:
- alias: "source"
bootstrapServers: strimzi-kafka-cluster-kafka-bootstrap.source.svc.cluster.local:9092
- alias: "target"
bootstrapServers: strimzi-kafka-cluster-kafka-bootstrap.target.svc.cluster.local:9092
mirrors:
- sourceCluster: "source"
targetCluster: "target"
topicsPattern: ".*"
groupsPattern: ".*"
sourceConnector:
tasksMax: 64
config:
auto.offset.reset: earliest
offset-syncs.topic.location: target
refresh.topics.interval.seconds: 10
topic.creation.default.replication.factor: 3
topic.creation.default.partitions: 1
sync.topic.acls.enabled: "false"
sync.topic.configs.enabled: "false"
refresh.topics.enabled: "false"
topic.creation.enable: "false"
heartbeatConnector:
config:
heartbeats.topic.replication.factor: 3
checkpointConnector:
tasksMax: 16
config:
sync.group.offsets.enabled: true
sync.group.offsets.interval.seconds: 2
emit.checkpoints.interval.seconds: 10
checkpoints.topic.replication.factor: 3
replication.policy.class: "org.apache.kafka.connect.mirror.DefaultReplicationPolicy"
offset-syncs.topic.location: target
I have explicitly tried setting the replication class but I assume that should have be the default.
The renaming is applied always only to the topics. The consumer group names remain the same regardless the replication policy. When syncing the offsets, the topics are renamed according to the policy as well, but the group is not.