apache-kafka

Cannot connect with kafka-ui to kafka

I start kafka locally with

$ docker run -p 9092:9092 apache/kafka:3.7.1
$ docker exec -it gifted_wright /bin/bash
e8bebba64b30:/$ cd /opt/kafka/bin
e8bebba64b30:/opt/kafka/bin$ sh kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-first-topic --partitions 1 --replication-factor 1
Created topic my-first-topic.
e8bebba64b30:/opt/kafka/bin$ sh kafka-topics.sh --bootstrap-server localhost:9092 --list
my-first-topic

I then can connect e.g. with the kafka plugin of IntelliJ Idea.

Then I start the kafka ui with

$ docker run -it -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true ghcr.io/kafbat/kafka-ui

When creating a cluster with only configureing bootstrap servers to be localhost:9092

9:44:45,536 INFO  [kafka-admin-client-thread | kafbat-ui-admin-1721382278-2] o.a.k.c.NetworkClient: [AdminClient clientId=kafbat-ui-admin-1721382278-2] Node -1 disconnected.
2024-07-19 09:44:45,537 WARN  [kafka-admin-client-thread | kafbat-ui-admin-1721382278-2] o.a.k.c.NetworkClient: [AdminClient clientId=kafbat-ui-admin-1721382278-2] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

Then a port fowarding is created which gives "Name does not resolve"

$ docker exec -it --user=root nice_wright /bin/sh
/ # apk add socat
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ncurses-terminfo-base (6.4_p20231125-r0)
(2/4) Installing libncursesw (6.4_p20231125-r0)
(3/4) Installing readline (8.2.1-r2)
(4/4) Installing socat (1.8.0.0-r0)
Executing busybox-1.36.1-r15.trigger
OK: 191 MiB in 26 packages
/ # socat tcp-listen:9092,fork tcp:host.docker.internal:9092

When clicking on validate in the ui I get

2024/07/19 09:42:59 socat[94] E getaddrinfo("host.docker.internal", "9092", {0x20,0,1,6}, {}): Name does not resolve

How to connect? Am I missing something central?

Solution

  • Could solve the issue with the hints in comment from OneCricketeer

    I created a network like explained on docker. Both started containers could ping each other both by hostname and IP-address. However, using bootstrap server competent_kare:9092 or its ip-address 172.18.0.2:9092 and click on validate in the ui I still got

    Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

    Commands which I used to verify the possibility of communication over the created network

    $ docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED              STATUS              PORTS                                       NAMES
17338825a793   ghcr.io/kafbat/kafka-ui   "/bin/sh -c 'java --…"   About a minute ago   Up About a minute   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   unruffled_hopper
0bc70820d669   apache/kafka:3.7.1        "/__cacert_entrypoin…"   11 minutes ago       Up 11 minutes       0.0.0.0:9092->9092/tcp, :::9092->9092/tcp   competent_kare

$ docker exec -it competent_kare /bin/bash
0bc70820d669:/$ ping unruffled_hopper
PING unruffled_hopper (172.18.0.3): 56 data bytes
64 bytes from 172.18.0.3: seq=0 ttl=42 time=0.188 ms
64 bytes from 172.18.0.3: seq=1 ttl=42 time=0.187 ms
^C
--- unruffled_hopper ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.181/0.185/0.188 ms
0bc70820d669:/$ ping 172.18.0.3
PING 172.18.0.3 (172.18.0.3): 56 data bytes
64 bytes from 172.18.0.3: seq=0 ttl=42 time=0.239 ms
64 bytes from 172.18.0.3: seq=1 ttl=42 time=0.240 ms
^C
--- 172.18.0.3 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.239/0.239/0.240 ms
0bc70820d669:/$ exit

$ docker exec -it unruffled_hopper /bin/sh
/ $ ping competent_kare
PING competent_kare (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: seq=0 ttl=42 time=0.095 ms
64 bytes from 172.18.0.2: seq=1 ttl=42 time=0.083 ms
^C
--- competent_kare ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.082/0.105/0.172 ms
/ $ ping 172.18.0.2
PING 172.18.0.2 (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: seq=0 ttl=42 time=0.265 ms
64 bytes from 172.18.0.2: seq=1 ttl=42 time=0.237 ms
^C
--- 172.18.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.237/0.251/0.265 ms
/ $ exit

    Using the following port forwarding solved the issue finally.

    $ docker exec -it --user=root unruffled_hopper /bin/sh
/ # apk add socat
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ncurses-terminfo-base (6.4_p20231125-r0)
(2/4) Installing libncursesw (6.4_p20231125-r0)
(3/4) Installing readline (8.2.1-r2)
(4/4) Installing socat (1.8.0.0-r0)
Executing busybox-1.36.1-r15.trigger
OK: 191 MiB in 26 packages
/ # socat tcp-listen:9092,fork tcp:competent_kare:9092

    I could in the GUI then connect via bootstrap server competent_kare:9092. The port forwarding seems to be necessary due to kafka returning a response that leads to a redirection to localhost:9092. So one solution seems to be to connect to the kafka host:port directly within the network. See as well Introduction to Apache Kafka.

    I hope I've got that right.

    Update

    A solution which sets advertised listeners can be found here. It seems in that way there is no need to create a shared docker network and link the containers.

    To run the docker compose file for the ui given in the post on linux I had to remove the volumes section at the end.

    Exchanging the image with apache/kafka:3.7.1 in the kafka compose file does not work since a org.apache.kafka.common.config.ConfigException: Missing required configuration zookeeper.connect which has no default value is returned on startup.