apache-kafkaapache-kafka-mirrormaker

How to decode messages in MM2 offset sync topic?


In reference to the offset sync topic covered in KIP-382 that maintains the cluster-to-cluster offset mapping, while consuming the messages from mm2-offset-syncs.target.internal found them to be serialized.

  1. Is there a way the output can be deserialized so its understandable using the kafka command line consumer?

     ./kafka-console-consumer.sh --bootstrap-server localhost:xxxx --topic mm2-offset-syncs.dest.internal --from-beginning       
    

Solution

  • Yes, you can use OffsetSyncFormatter to deserialize the content of your offset syncs topics. For example:

    ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
      --topic mm2-offset-syncs.target.internal \
      --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter \
      --from-beginning
    

    The more details, see KIP-597: MirrorMaker2 internal topics Formatters