apache-kafkaapache-kafka-connectapache-kafka-mirrormaker

where are the kafka mirrormaker2 consumer groups


I have added a mirror maker 2 source connector to my connect cluster:

{
  "name": "test_mirror",
  "config": {
    "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "tasks.max": 1,
    "source.cluster.alias": "thirdparty",
    "target.cluster.alias": "kafka",
    "source.cluster.bootstrap.servers": "thirdparty:9092",
    "target.cluster.bootstrap.servers": "kafka:9092",
    "topics": "test",
    "sync.topic.acls.enabled": false,
    "refresh.topics.enabled": false,
    "refresh.groups.enabled": false,
    "offset-syncs.topic.replication.factor": 1,
    "replication.factor": 1
  }
}

This works as expected, I can consume the topic thirdparty.test, on my cluster.

However, when I inspect both clusters. I do don't see any consumer groups related with the mirror.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --all-groups --describe

How does this work? How can it replicate data without creating consumer groups on source and/or target?


Solution

  • Kafka Connect source connectors store offsets in their configured offsets topic, not in the __consumer_offsets topic. Only sink connectors use consumer groups.

    But that's unrelated to syncing of external groups. More importantly, if MirrorMaker2 committed to the __consumer_offsets topic, it'd need extra logic to filter its own offsets while mirroring other groups