I am having Kafka consumer (zookeeper based) with client version 0.8.1.1 which uses zookeeper as offset storage (consuming from Kafka cluster 1.0.1). Now, I want to replicate the messages and offsets (that are stored in zookeeper) to a new Kafka cluster 2.8.0 using MirrorMaker2. But I need the replicated offsets to store in Kafka storage instead of zookeeper storage. From my testing, I found that MirrorMaker2 is not replicating the offsets that are stored in zookeeper storage but it does only if it is stored in Kafka storage.
One of the options is to upgrade the consumer to 0.8.2.x and set dual.commit.enabled to True so that MirrorMaker2 can replicate the offset from Kafka storage to Kafka storage. But for some reason, I would like to avoid this due to technical limitations.
Is there any config or workaround to achieve the use case to migrate the offset from zookeeper storage to Kafka storage using MirrorMaker2?
To my knowledge, MM2 was written without any Zookeeper API in mind, as there were already talks about removing Zookeeper completely when it was developed, so there was no reason to add functions for it.
In fact, I'm not even sure it'll work on Kafka clusters less than approx 0.10.2, as it uses other metadata to sync topic information using AdminClient API and record timestamps (and possibly headers, added in 0.11)