I am trying to configure two Kafka clusters to connect to the same Schema Registry. Assume everything is reachable network wise. Since Schema Registry needs Kafka's bootstrap servers (and possibly other Kafka related properties from both clusters), how can I configure everything so that this Schema Registry connects to both Kafka clusters?.
Consider this example using docker-compose:
networks:
local_kafka:
name: local_kafka
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.5.2
container_name: zookeeper
networks:
- local_kafka
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=*" # For easier local debugging of zookeeper issues.
zookeeper2:
image: confluentinc/cp-zookeeper:7.5.2
container_name: zookeeper2
networks:
- local_kafka
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
KAFKA_OPTS: "-Dzookeeper.4lw.commands.whitelist=*" # For easier local debugging of zookeeper issues.
broker1:
image: confluentinc/cp-kafka:7.5.2
container_name: broker1
networks:
- local_kafka
ports:
- "19092:19092"
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_BOOTSTRAP_SERVERS: broker1:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092,CONNECTIONS_FROM_HOST://localhost:19092
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_MIN_INSYNC_REPLICAS: 1
KAFKA_SCHEMA_REGISTRY_URL: schemaregistry:8081
ALLOW_PLAINTEXT_LISTENERS: "true"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_RETRIES: 3
broker2:
image: confluentinc/cp-kafka:7.5.2
container_name: broker2
networks:
- local_kafka
ports:
- "19093:19093"
depends_on:
- zookeeper2
environment:
KAFKA_BROKER_ID: 2
KAFKA_BOOTSTRAP_SERVERS: broker2:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper2:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092,CONNECTIONS_FROM_HOST://localhost:19093
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_MIN_INSYNC_REPLICAS: 1
KAFKA_SCHEMA_REGISTRY_URL: schemaregistry:8081
ALLOW_PLAINTEXT_LISTENERS: "true"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_RETRIES: 3
schemaregistry:
image: confluentinc/cp-schema-registry:7.5.2
container_name: schemaregistry
networks:
- local_kafka
environment:
SCHEMA_REGISTRY_HOST_NAME: schemaregistry
SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8081"
SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: PLAINTEXT
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: <<<<------ WHAT TO FILL HERE?
SCHEMA_REGISTRY_DEBUG: "true"
SCHEMA_REGISTRY_KAFKASTORE_TOPIC: _registry-schemas
ports:
- "8081:8081"
broker1 is one Kafka, broker2 is a different Kafka. I wanted both connected to schemaregistry
.
Is this possible?
Thank you.
It's not possible since the schema registry creates one _registry-schemas
topic, by itself, on one kafka cluster (bootstrap severs of one cluster).
The connection is one-way, and the broker has no relationship with the Registry. There's no broker configuration for schema.registry.url
.
You'd need distinct Registry servers for different clusters