kubernetesapache-kafkaapache-kafka-mirrormaker

Copy data from one Kafka cluster to another in different Kubernetes cluster


I have two Kubernetes clusters located on two different networks. On each k8s cluster, a Kafka cluster is deployed using Apache Kafka packaged by Bitnami.

I would like to copy data from one Kafka cluster to another. I found that MirrorMaker can be used for this, but I'm not sure how to connect two Kubernetes clusters without exposing the Kafka cluster to the internet. Do you have any ideas or advice on safely transferring kafka data from one cluster to another ?


Solution

  • You're going to need some network connection between the clusters. This does not imply "the internet", but that's also not a problem related to Kafka or Kubernetes other than you'll need an Ingress resource and TLS connection to perform external cluster TCP Kafka Connections.

    Yes, MirrorMaker2 or other Kafka "linking/replicating" solutions is exactly what you want

    It is worth pointing out that Strimzi already offers MirrorMaker2, and would completely replace your Bitnami solution