(I know this question is similar to this other one, but the other one uses Zookeeper which is deprecated.)
I have this simple docker-compose:
services:
kafka:
container_name: dotnetbus-kafka
image: apache/kafka:3.8.0
ports:
- 9092:9092
init-kafka:
container_name: dotnetbus-init-kafka
image: apache/kafka:3.8.0
depends_on:
- kafka
command: sleep 1000
Eventually, I want to put a create-topic command in the init-kafka
images instead of the sleep
, but first I need to get those commands to actually work.
When I shell into the init-kafka
image and try to list topics, this happens:
$ /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:9092 --list
[2024-09-10 03:17:03,924] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-09-10 03:17:04,027] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
...
(repeats infinitely)
I've tried a bunch of variations of that command. I must be understanding docker wrong, or Kafka wrong, or both.
However, when I run a similar command from my host machine (to localhost:9092) it works fine.
What am I missing? Why can't I get init-kafka
to reach kafka
?
With some help from the comments and this example, I've got it working.
For KRaft, the compose environment needs to configure a "controller" listener, in addition to any other listeners.
Kafka will treat the config as Zookeeper unless it sees KAFKA_CONTROLLER_QUORUM_VOTERS
.
These last 4 vars on my broker image are absolutely necessary. I honestly don't know what they do right now, but the default values didn't work (subscriptions always failed), and the error messages I was getting were 100% not-helpful for me with my present knowledge:
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
The following compose config configures 3 listeners:
The kafka1
service is the broker, and kafka-init
creates topics on it.
services:
kafka1:
container_name: dotnetbus-kafka
image: apache/kafka:3.8.0
ports:
- 9092:9092
environment:
KAFKA_NODE_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,INTERBROKER:PLAINTEXT,HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'HOST://localhost:9092,INTERBROKER://kafka1:19092'
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:29092'
KAFKA_LISTENERS: 'HOST://:9092,INTERBROKER://:19092,CONTROLLER://:29092'
KAFKA_INTER_BROKER_LISTENER_NAME: 'INTERBROKER'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
kafka-init:
container_name: dotnetbus-init-kafka
image: apache/kafka:3.8.0
depends_on:
- kafka1
command:
- /bin/sh
- -c
- |
sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list
echo -e 'Creating kafka topics'
sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-request --replication-factor 1 --partitions 1
sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-status --replication-factor 1 --partitions 1
echo -e 'Successfully created the following topics:'
sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list
(Note: The depends_on
ought to accompanied by a good health check, which I clearly haven't done in this config.)