dockerapache-kafkadocker-composekafka-cluster

Kafdrop - Cannot connect to Kafka Cluster setup using bitnami/kafka


I setup a kafka cluster using bitnami kafka and zookeeper and I wanted to view this cluster or at least one broker using kafdrop. I used docker compose to build all the components. I initially followed this tutorial and then added the kafdrop config in the docker-compose.yml

version: '2'

networks:
  kafka-net:
    driver: bridge

services:
  zookeeper-server:
    image: 'bitnami/zookeeper:latest'
    networks:
      - kafka-net
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafdrop:
    image: obsidiandynamics/kafdrop
    networks:
      - kafka-net
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "PLAINTEXT://localhost:9092,PLAINTEXT://localhost:9093,PLAINTEXT://localhost:9094"
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
    depends_on:
      - "kafka-server1"
      - "kafka-server2"
      - "kafka-server3"
  kafka-server1:
    image: 'bitnami/kafka:latest'
    networks:
      - kafka-net    
    ports:
      - '9092:9092'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper-server
  kafka-server2:
    image: 'bitnami/kafka:latest'
    networks:
      - kafka-net    
    ports:
      - '9093:9092'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9093
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper-server
  kafka-server3:
    image: 'bitnami/kafka:latest'
    networks:
      - kafka-net    
    ports:
      - '9094:9092'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9094
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper-server

My main issue is that kafdrop always throw this error:

020-08-26 10:53:53.517  WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient                    : [AdminClient clientId=kafdrop-admin] Connection to node -3 (localhost/127.0.0.1:9094) could not be established. Broker may not be available.
2020-08-26 10:53:53.522  WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient                    : [AdminClient clientId=kafdrop-admin] Connection to node -2 (localhost/127.0.0.1:9093) could not be established. Broker may not be available.
2020-08-26 10:53:53.526  WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient                    : [AdminClient clientId=kafdrop-admin] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2020-08-26 10:53:53.627  WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient                    : [AdminClient clientId=kafdro

I have tried changing the value of KAFKA_BROKERCONNECT with the ff values but all didn't work.

I am actually just guessing the correct config syntax so any explanation with this one is appreciated :).

Also, is adding networks property needed on kafdrop config? Kafdrop have sample docker-compose file and this one doesn't have network config so I wonder why/if network is needed.


Solution

  • Your second way is the right way. Also for the KAFKA_CFG_ADVERTISED_LISTENERS vars which I'm not sure are necessary. You just need to make sure to use the right ports. This should work fine:

    version: '2'
    
    networks:
      kafka-net:
        driver: bridge
    
    services:
      zookeeper-server:
        image: 'bitnami/zookeeper:latest'
        networks:
          - kafka-net
        ports:
          - '2181:2181'
        environment:
          - ALLOW_ANONYMOUS_LOGIN=yes
      kafdrop:
        image: obsidiandynamics/kafdrop
        networks:
          - kafka-net
        restart: "no"
        ports:
          - "9000:9000"
        environment:
          KAFKA_BROKERCONNECT: "PLAINTEXT://kafka-server1:9092,PLAINTEXT://kafka-server2:9092,PLAINTEXT://kafka-server3:9092"
          JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
        depends_on:
          - "kafka-server1"
          - "kafka-server2"
          - "kafka-server3"
      kafka-server1:
        image: 'bitnami/kafka:latest'
        networks:
          - kafka-net    
        ports:
          - '9092:9092'
        environment:
          - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
          - ALLOW_PLAINTEXT_LISTENER=yes
        depends_on:
          - zookeeper-server
      kafka-server2:
        image: 'bitnami/kafka:latest'
        networks:
          - kafka-net    
        ports:
          - '9093:9092'
        environment:
          - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
          - ALLOW_PLAINTEXT_LISTENER=yes
        depends_on:
          - zookeeper-server
      kafka-server3:
        image: 'bitnami/kafka:latest'
        networks:
          - kafka-net    
        ports:
          - '9094:9092'
        environment:
          - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
          - ALLOW_PLAINTEXT_LISTENER=yes
        depends_on:
          - zookeeper-server