dockerapache-kafkaconfluent-schema-registryksqldb

Kafka KSQL Docker cant connect to Schema Registry


I'm trying to create a stream in my local kafka integration however KSQL server can't connect to the schema registry or that parameter never reaches the KSQL server container. While the KSQL server is starting I can see in the logs that the ksql.schema.registry.url = is always empty. As per documentation, this should not be updated using the KSQL client. I have tried going into the docker and adding the line directly into the configuration file but it didn't work.

version: '2'
services:
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=kafka
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENERS=PLAINTEXT://:29092,EXTERNAL://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,EXTERNAL://kafka:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_SCHEMA_REGISTRY_URL=schemaregistry:8085
    platform: linux/amd64
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=zookeeper
      - ZOOKEEPER_CLIENT_PORT=2181
      - ZOOKEEPER_TICK_TIME=2000
    platform: linux/amd64
   
  schemaregistry:
    image: confluentinc/cp-schema-registry:6.2.0
    restart: always
    depends_on:
      - zookeeper
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181"
      SCHEMA_REGISTRY_HOST_NAME: schemaregistry
      SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8085"
    ports:
      - 8085:8085
    platform: linux/amd64

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.29.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - kafka
      - schemaregistry
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: kafka:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_METRICS_ENABLED: "false"
      KSQL_SUPPORT_METRICS_ENABLE: "false"
      KSQL_SCHEMA_REGISTRY_URL: "http://schemaregistry:8085"
    platform: linux/amd64

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.29.0
    container_name: ksqldb-cli
    depends_on:
      - kafka
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true
    platform: linux/amd64

This is how I am trying to create it and the response:

ksql> CREATE STREAM youtube_videos (video_id VARCHAR KEY, title VARCHAR, views INTEGER, likes INTEGER, comments INTEGER) WITH (KAFKA_TOPIC='youtube_videos', PARTITIONS=1, VALUE_FORMAT='avro');
Cannot create topic 'youtube_videos' with format AVRO without configuring 'ksql.schema.registry.url'

Solution

  • The correct variable is KSQL_KSQL_SCHEMA_REGISTRY_URL. https://github.com/confluentinc/cp-all-in-one/blob/v7.7.1/cp-all-in-one-kraft/docker-compose.yml#L114

    Also, key points

    1. Don't mix/match Confluent versions. Your schema registry image tag should match kafka, at a minimum.
    2. The broker doesn't need a schema registry connection. KAFKA_SCHEMA_REGISTRY_URL is not a valid broker config. And KAFKA_ADVERTISED_HOST_NAME isn't a valid Zookeeper config... That property itself has been deprecated from Kafka, so remove it there. It would be recommended to consult official documentation for each environment variable you've tried to configure.
    3. Schema registry's Zookeeper connection mode has been deprecated, if not removed, for years. Connect to the broker instead for kafka store connection variable (see linked compose file).
    4. You've setup KAFKA_ADVERTISED_LISTENERS as two "docker internal" hostnames. The "external" listener should use localhost:9092