apache-kafkaapache-kafka-mirrormaker

Mirrormaker 2: different partition count


I am using MM2 to replicate clusters. The current topics are overprovisioned (too many partitions) and I need to reduce them as part of the migration to fit actual needs & confluent cloud restrictions.

Is this at all possible? I see that this helps with offset replication. Is it maybe possible to replicate without offset replication to allow replicating into a reduced number of partitions?


Solution

  • We faced the same issue and our solution is to use MirrorMaker1(kafka-mirror-maker.sh).
    MirrorMaker1 uses Consumer/Producer API for replication and doesn't require partition aligning of source/target topics instead of MM2.
    The documentation of fresh Kafka versions 2.8+ doesn't mention MM1, I found docs about it in previous versions of Kafka documentation, e.g. apache kafka 2.0.