dockerapache-kafkaapache-kafka-mirrormaker

Kafka - MirrorMaker 2.0 production deployment


Question: How to Configure Docker for Kafka MirrorMaker 2.0 to Replicate Messages Between Two Clusters?

I have two Kafka clusters: a source cluster and a destination cluster, each with 3 brokers. I want to replicate messages from the source cluster to the destination cluster using MirrorMaker 2.0, and I'm using the official Apache Kafka Docker image for this setup.

Could someone help me with the configuration of the docker-compose.yml file?

  1. Should I run the MirrorMaker command each time the Docker container starts, or is there a better way to configure this for production use?
  2. Is it sufficient to run just one instance of MirrorMaker 2.0, or should I set up a dedicated MirrorMaker cluster with 3 nodes, similar to my Kafka clusters?

I'm looking for the best practices for running MirrorMaker 2.0 in a production environment using Docker.

I have tried to run it inside the docker container with below docker compose file

version: '3.8'
services:
  mirrormaker2:
    image: apache/kafka:latest
    hostname: mirrormaker2
    container_name: mirrormaker2
    volumes:
      - ./connect-mirror-maker.properties:/opt/kafka/config/connect-mirror-maker.properties:rw
    network_mode: "host"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@:9093'
      KAFKA_LISTENERS: 'PLAINTEXT://:19092,CONTROLLER://10.142.13.108:9093,PLAINTEXT_HOST://:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://:19092,PLAINTEXT_HOST://:29092'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
    command: >
      sh -c "cd /opt/kafka && ./bin/connect-mirror-maker.sh ./config/connect-mirror-maker.properties"

Solution

  • Should I run the MirrorMaker command each time the Docker container starts

    You'll have to. Otherwise, it'll start a broker, by default.

    is there a better way to configure this for production use?

    Yes. Run a container that starts Kafka Connect server rather than a broker. Otherwise none of the Kafka Connect / MirrorMaker2 properties will be set other than those mounted from your file, and the variables that you have set are not necessary.

    confluentinc/cp-kafka-connect is one example ; I maintain my own.

    Then you'd use the Connect REST API to configure and start the replication process rather than mounting an external volume or calling an external script.


    One instance will work, but more than be added for scalability.

    Docker Compose generally is replaced by a Kubernetes cluster in production, so you could use Strimzi to manage this process better