apache-kafkaapache-kafka-mirrormaker

Is it possible to replicate kafka topics without alias prefix with MirrorMaker2


I'm trying to set up replication between 2 clusters but don't want want the topic names to be changed. for example if i have a topic called "some_topic" it is automatically replicated to "cluster1.some_topic", I'm pretty sure this can be done but haven't found the correct config to change this

My current config "mirrormaker2.properties"

# Sample MirrorMaker 2.0 top-level configuration file
# Run with ./bin/connect-mirror-maker.sh connect-mirror-maker.properties 

# specify any number of cluster aliases
clusters = cluster1, cluster2

# connection information for each cluster
cluster1.bootstrap.servers = host1:9092,host2:9092,host3:9092
cluster2.bootstrap.servers = rep_host1:9092,rep_host2:9092,rep_host3:9092

# enable and configure individual replication flows
cluster1->cluster2.enabled = true
cluster1->cluster2.topics = sometopic.*

# customize as needed
# replication.policy.separator = _
# sync.topic.acls.enabled = false
# emit.heartbeats.interval.seconds = 5

for reference:


Solution

  • I think the answer above is inappropriate.

    In Mirror Maker 2.0, if you want to keep the topic unmodified, you have to implement ReplicationPolicy.

    You can refer to DefaultReplicationPolicy.class, and then override formatRemoteTopic(), after that you have to remove sourceClusterAlias + separator. In the end, configure replication.policy.class in the mm2.properties

    I defined MigrationReplicationPolicy.class

    replication.policy.class = org.apache.kafka.connect.mirror.MigrationReplicationPolicy
    

    You should see MirrorClientConfig,class, I know that you will understand