apache-kafkaconfluent-schema-registryapache-kafka-mirrormaker

How to replicate schema with Kafka mirror maker?


We are using the mirror maker to sync on-premise and AWS Kafka topics. How can a topic with its schema registered in on-premise be replicated exactly the same in other clusters (AWS in this case)? How Avro schema is replicated using mirror maker?


Solution

  • MirrorMaker only copies byte arrays, not schemas. And doesn't care about the format of the data

    As of Confluent 4.x or later, Schema Registry added endpoint GET /schemas/ids/(number). So, if your consumers are configured to the original registry, this shouldn't matter since your destination consumers can lookup the schema ID.

    You otherwise can mirror the _schemas topic as well, as recommend by Confluent when using Confluent Replicator

    If you absolutely need one-to-one schema copying, you would need to implement a MessageHandler interface, and pass this on to the MirrorMaker command, to get and post the schema, similar to the internal logic I added to this Kafka Connect plugin (which you could use Connect instead of MirrorMaker). https://github.com/OneCricketeer/schema-registry-transfer-smt