I have added a mirror maker 2 source connector to my connect cluster:
{
"name": "test_mirror",
"config": {
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"tasks.max": 1,
"source.cluster.alias": "thirdparty",
"target.cluster.alias": "kafka",
"source.cluster.bootstrap.servers": "thirdparty:9092",
"target.cluster.bootstrap.servers": "kafka:9092",
"topics": "test",
"sync.topic.acls.enabled": false,
"refresh.topics.enabled": false,
"refresh.groups.enabled": false,
"offset-syncs.topic.replication.factor": 1,
"replication.factor": 1
}
}
This works as expected, I can consume the topic thirdparty.test
, on my cluster.
However, when I inspect both clusters. I do don't see any consumer groups related with the mirror.
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --describe
How does this work? How can it replicate data without creating consumer groups on source and/or target?
Kafka Connect source connectors store offsets in their configured offsets topic, not in the __consumer_offsets topic. Only sink connectors use consumer groups.
But that's unrelated to syncing of external groups. More importantly, if MirrorMaker2 committed to the __consumer_offsets topic, it'd need extra logic to filter its own offsets while mirroring other groups