apache-kafka

Can't acces Kafka with distant host

I try to access Kafka server (10.10.45.91) from a distant host (10.10.45.90).

VMs Alma Linux 9.x, Kafka Kraft 2.13-3.7.1 on both server and distant host.

Note: thanks to @OneCricketeer I've discover kcat. I've rewrite the question to be more precise.

What I did:

On the server VM (10.10.45.91):

I've add a topic logstash-topic

FW: open 9092 port

Set configuration:

sudo nano /opt/kafka/config/server.properties

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://10.10.45.91:9092

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 10.10.45.91:9092 --describe --topic logstash-topic

Works as expected

On the distant host 10.10.45.90:

nc -vz 10.10.45.91 9092 => ok

kcat -L -b 10.10.45.91:9092

Metadata for all topics (from broker -1: 10.10.45.91:9092/bootstrap):
 1 brokers:
  broker 1 at localhost:9092 (controller)
 4 topics:
  topic "logstash2" with 1 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1
  topic "quickstart-events" with 1 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1
  topic "__consumer_offsets" with 50 partitions:
...
 topic "logstash-topic" with 1 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1

kcat -b 10.10.45.91:9092 -t logstash-topic

% Auto-selecting Consumer mode (use -P or -C to override)
%3|1724217508.095|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect Connection refused (after 2ms in state CONNECT)
% ERROR: Local: Broker transport failure: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 fai2ms in state CONNECT)
%3|1724217508.212|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/1]: localhost:9092/1: Connect Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
% ERROR: Local: Broker transport failure: localhost:9092/1: Connect to ipv4#127.0.0.1:9092 fai0ms in state CONNECT, 1 identical error(s) suppressed)

It seems that kafka server send back loopback IP (as the main broker) which make kcat to query himself.

Note: When I run a logstash with correct configuration (that work well on server VM), Logstash seems to "query" to localhost, as well.

Seems I got an error on my Kafka config file.

I've try a lot of configuration on field listeners, advertised.listeners and listener.security.protocol.map field.

Maybe something that not work with IP address but with hostname, but I can't find out.

2 days spends on this problem, reading posts and S.O., now I need help :).

Solution

  • I've modified the wrong properties file, so Kafka can't apply my modifications.

    Why:

    Kafka (installed with Kraf) got 2 server.properties:

    /opt/kafka/config/server.properties
/opt/kafka/config/kraft/server.properties

    I've set-up Kafka as a daemon: /etc/systemd/system/kafka-server.service

    ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties

    Finally, following documentation, I've set up my Kafka config in /opt/kafka/config/server.properties which will never be applied.

    Thanks to @OneCricketeer, I've check my daemon config and find out.

    Adding the conf in the correct properties file make kafka work perfectly.

    Thanks for the help and kcat tool.

    Hope that could help.