apache-kafkaapache-kafka-mirrormaker

How to deserialize avro message using mirrormaker?


I want to replicate a kafka topic to an azure event hub. The messages are in avro format and uses a schema that is behind a schema registry with USER_INFO authentication.

Using a java client to connect to kafka, I can use a KafkaAvroDeserializer to deserialize the message correctly.

But this configuration doesn't seems to work with mirrormaker.

Is is possible to deserialize the avro message using mirrormaker before sending it ?

Cheers


Solution

  • For MirrorMaker1, the consumer deserializer properties are hard-coded

    Unless you plan on re-serializing the data into a different format when the producer sends data to EventHub, you should stick to using the default ByteArrayDeserializer.

    If you did want to manipulate the messages in any way, that would need to be done with a MirrorMakerMessageHandler subclass


    For MirrorMaker2, you can use AvroConverter followed by some transforms properties, but still ByteArrayConverter would be preferred for a one-to-one byte copy.