apache-kafkadocker-compose

In docker-compose, my init-kafka image is failing to list topics on my kafka image. How do I fix this? (With KRaft, not Zookeeper!)


(I know this question is similar to this other one, but the other one uses Zookeeper which is deprecated.)

I have this simple docker-compose:

services:
  kafka:
    container_name: dotnetbus-kafka
    image: apache/kafka:3.8.0
    ports:
      - 9092:9092

  init-kafka:
    container_name: dotnetbus-init-kafka
    image: apache/kafka:3.8.0
    depends_on:
      - kafka
    command: sleep 1000

NOTE: I'm looking for a solution that uses KRaft, not Zookeeper (because the latter is deprecated).

Eventually, I want to put a create-topic command in the init-kafka images instead of the sleep, but first I need to get those commands to actually work.

When I shell into the init-kafka image and try to list topics, this happens:

$ /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:9092 --list
[2024-09-10 03:17:03,924] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-09-10 03:17:04,027] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
...
(repeats infinitely)

I've tried a bunch of variations of that command. I must be understanding docker wrong, or Kafka wrong, or both.

However, when I run a similar command from my host machine (to localhost:9092) it works fine.

What am I missing? Why can't I get init-kafka to reach kafka?


Solution

  • With some help from the comments and this example, I've got it working.

    Things I learned:

    My working compose config

    The following compose config configures 3 listeners:

    The kafka1 service is the broker, and kafka-init creates topics on it.

    services:
      kafka1:
        container_name: dotnetbus-kafka
        image: apache/kafka:3.8.0
        ports:
          - 9092:9092
        environment:
          KAFKA_NODE_ID: 1
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,INTERBROKER:PLAINTEXT,HOST:PLAINTEXT'
          KAFKA_ADVERTISED_LISTENERS: 'HOST://localhost:9092,INTERBROKER://kafka1:19092'
          KAFKA_PROCESS_ROLES: 'broker,controller'
          KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka1:29092'
          KAFKA_LISTENERS: 'HOST://:9092,INTERBROKER://:19092,CONTROLLER://:29092'
          KAFKA_INTER_BROKER_LISTENER_NAME: 'INTERBROKER'
          KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
          KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
          KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
          KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
    
      kafka-init:
        container_name: dotnetbus-init-kafka
        image: apache/kafka:3.8.0
        depends_on:
          - kafka1
        command:
          - /bin/sh
          - -c
          - |
            sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list
            echo -e 'Creating kafka topics'
            sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-request --replication-factor 1 --partitions 1
            sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --create --if-not-exists --topic trade-status --replication-factor 1 --partitions 1
            echo -e 'Successfully created the following topics:'
            sh /opt/kafka/bin/kafka-topics.sh --bootstrap-server dotnetbus-kafka:19092 --list
    

    (Note: The depends_on ought to accompanied by a good health check, which I clearly haven't done in this config.)