apache-kafkaconfluent-schema-registry

Multiple Kafka Clusters connected to the same Schema Registry


I am trying to configure two Kafka clusters to connect to the same Schema Registry. Assume everything is reachable network wise. Since Schema Registry needs Kafka's bootstrap servers (and possibly other Kafka related properties from both clusters), how can I configure everything so that this Schema Registry connects to both Kafka clusters?.

Consider this example using docker-compose:

networks:
  local_kafka:
    name: local_kafka

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.2
    container_name: zookeeper
    networks:
      - local_kafka
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=*" # For easier local debugging of zookeeper issues.

  zookeeper2:
    image: confluentinc/cp-zookeeper:7.5.2
    container_name: zookeeper2
    networks:
      - local_kafka
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=*" # For easier local debugging of zookeeper issues.

  broker1:
    image: confluentinc/cp-kafka:7.5.2
    container_name: broker1
    networks:
      - local_kafka
    ports:
      - "19092:19092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_BOOTSTRAP_SERVERS: broker1:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092,CONNECTIONS_FROM_HOST://localhost:19092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_MIN_INSYNC_REPLICAS: 1
      KAFKA_SCHEMA_REGISTRY_URL: schemaregistry:8081
      ALLOW_PLAINTEXT_LISTENERS: "true"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_RETRIES: 3

  broker2:
    image: confluentinc/cp-kafka:7.5.2
    container_name: broker2
    networks:
      - local_kafka
    ports:
      - "19093:19093"
    depends_on:
      - zookeeper2
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_BOOTSTRAP_SERVERS: broker2:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper2:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092,CONNECTIONS_FROM_HOST://localhost:19093
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_MIN_INSYNC_REPLICAS: 1
      KAFKA_SCHEMA_REGISTRY_URL: schemaregistry:8081
      ALLOW_PLAINTEXT_LISTENERS: "true"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_RETRIES: 3

  schemaregistry:
    image: confluentinc/cp-schema-registry:7.5.2
    container_name: schemaregistry
    networks:
      - local_kafka
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schemaregistry
      SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8081"
      SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: PLAINTEXT
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: <<<<------ WHAT TO FILL HERE?
      SCHEMA_REGISTRY_DEBUG: "true"
      SCHEMA_REGISTRY_KAFKASTORE_TOPIC: _registry-schemas
    ports:
      - "8081:8081"

broker1 is one Kafka, broker2 is a different Kafka. I wanted both connected to schemaregistry.

Is this possible?

Thank you.


Solution

  • It's not possible since the schema registry creates one _registry-schemas topic, by itself, on one kafka cluster (bootstrap severs of one cluster).

    The connection is one-way, and the broker has no relationship with the Registry. There's no broker configuration for schema.registry.url.

    You'd need distinct Registry servers for different clusters