I setup a kafka cluster using bitnami kafka and zookeeper and I wanted to view this cluster or at least one broker using kafdrop. I used docker compose to build all the components. I initially followed this tutorial and then added the kafdrop config in the docker-compose.yml
version: '2'
networks:
kafka-net:
driver: bridge
services:
zookeeper-server:
image: 'bitnami/zookeeper:latest'
networks:
- kafka-net
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafdrop:
image: obsidiandynamics/kafdrop
networks:
- kafka-net
restart: "no"
ports:
- "9000:9000"
environment:
KAFKA_BROKERCONNECT: "PLAINTEXT://localhost:9092,PLAINTEXT://localhost:9093,PLAINTEXT://localhost:9094"
JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
depends_on:
- "kafka-server1"
- "kafka-server2"
- "kafka-server3"
kafka-server1:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9092:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
kafka-server2:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9093:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9093
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
kafka-server3:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9094:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9094
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
My main issue is that kafdrop always throw this error:
020-08-26 10:53:53.517 WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient : [AdminClient clientId=kafdrop-admin] Connection to node -3 (localhost/127.0.0.1:9094) could not be established. Broker may not be available.
2020-08-26 10:53:53.522 WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient : [AdminClient clientId=kafdrop-admin] Connection to node -2 (localhost/127.0.0.1:9093) could not be established. Broker may not be available.
2020-08-26 10:53:53.526 WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient : [AdminClient clientId=kafdrop-admin] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2020-08-26 10:53:53.627 WARN 1 [| kafdrop-admin] o.a.k.c.NetworkClient : [AdminClient clientId=kafdro
I have tried changing the value of KAFKA_BROKERCONNECT
with the ff values but all didn't work.
I am actually just guessing the correct config syntax so any explanation with this one is appreciated :).
Also, is adding networks
property needed on kafdrop config? Kafdrop have sample docker-compose file and this one doesn't have network config so I wonder why/if network
is needed.
Your second way is the right way. Also for the KAFKA_CFG_ADVERTISED_LISTENERS
vars which I'm not sure are necessary. You just need to make sure to use the right ports. This should work fine:
version: '2'
networks:
kafka-net:
driver: bridge
services:
zookeeper-server:
image: 'bitnami/zookeeper:latest'
networks:
- kafka-net
ports:
- '2181:2181'
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafdrop:
image: obsidiandynamics/kafdrop
networks:
- kafka-net
restart: "no"
ports:
- "9000:9000"
environment:
KAFKA_BROKERCONNECT: "PLAINTEXT://kafka-server1:9092,PLAINTEXT://kafka-server2:9092,PLAINTEXT://kafka-server3:9092"
JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
depends_on:
- "kafka-server1"
- "kafka-server2"
- "kafka-server3"
kafka-server1:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9092:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
kafka-server2:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9093:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server
kafka-server3:
image: 'bitnami/kafka:latest'
networks:
- kafka-net
ports:
- '9094:9092'
environment:
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181
- ALLOW_PLAINTEXT_LISTENER=yes
depends_on:
- zookeeper-server