I'm trying to create a stream in my local kafka integration however KSQL server can't connect to the schema registry or that parameter never reaches the KSQL server container.
While the KSQL server is starting I can see in the logs that the ksql.schema.registry.url =
is always empty. As per documentation, this should not be updated using the KSQL client. I have tried going into the docker and adding the line directly into the configuration file but it didn't work.
version: '2'
services:
kafka:
image: confluentinc/cp-kafka:latest
container_name: kafka
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=kafka
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENERS=PLAINTEXT://:29092,EXTERNAL://:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:29092,EXTERNAL://kafka:9092
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
- KAFKA_SCHEMA_REGISTRY_URL=schemaregistry:8085
platform: linux/amd64
depends_on:
- zookeeper
zookeeper:
image: confluentinc/cp-zookeeper:latest
container_name: zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper
- ZOOKEEPER_CLIENT_PORT=2181
- ZOOKEEPER_TICK_TIME=2000
platform: linux/amd64
schemaregistry:
image: confluentinc/cp-schema-registry:6.2.0
restart: always
depends_on:
- zookeeper
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181"
SCHEMA_REGISTRY_HOST_NAME: schemaregistry
SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8085"
ports:
- 8085:8085
platform: linux/amd64
ksqldb-server:
image: confluentinc/ksqldb-server:0.29.0
hostname: ksqldb-server
container_name: ksqldb-server
depends_on:
- kafka
- schemaregistry
ports:
- "8088:8088"
environment:
KSQL_LISTENERS: http://0.0.0.0:8088
KSQL_BOOTSTRAP_SERVERS: kafka:29092
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
KSQL_METRICS_ENABLED: "false"
KSQL_SUPPORT_METRICS_ENABLE: "false"
KSQL_SCHEMA_REGISTRY_URL: "http://schemaregistry:8085"
platform: linux/amd64
ksqldb-cli:
image: confluentinc/ksqldb-cli:0.29.0
container_name: ksqldb-cli
depends_on:
- kafka
- ksqldb-server
entrypoint: /bin/sh
tty: true
platform: linux/amd64
This is how I am trying to create it and the response:
ksql> CREATE STREAM youtube_videos (video_id VARCHAR KEY, title VARCHAR, views INTEGER, likes INTEGER, comments INTEGER) WITH (KAFKA_TOPIC='youtube_videos', PARTITIONS=1, VALUE_FORMAT='avro');
Cannot create topic 'youtube_videos' with format AVRO without configuring 'ksql.schema.registry.url'
The correct variable is KSQL_KSQL_SCHEMA_REGISTRY_URL
. https://github.com/confluentinc/cp-all-in-one/blob/v7.7.1/cp-all-in-one-kraft/docker-compose.yml#L114
Also, key points
KAFKA_SCHEMA_REGISTRY_URL
is not a valid broker config. And KAFKA_ADVERTISED_HOST_NAME
isn't a valid Zookeeper config... That property itself has been deprecated from Kafka, so remove it there. It would be recommended to consult official documentation for each environment variable you've tried to configure.KAFKA_ADVERTISED_LISTENERS
as two "docker internal" hostnames. The "external" listener should use localhost:9092