[SOLVED] Problem with ADVERTISED

Problem with ADVERTISED_LISTENER on macos

I start kafka with this docker-compose.yml on my Mac:

 mydb:
 [...]
   environment: 
     kafka_bootstrap_servers: kafka:9093

 kafka:
    image: wurstmeister/kafka:2.13-2.6.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_CREATE_TOPICS: "mytopic:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      #KAFKA_LISTENERS: PLAINTEXT://:9092
      #KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092

      KAFKA_LISTENERS: LISTENER_INTERN://:9093,LISTENER_EXTERN://127.0.0.1:9092
      KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERN://:9093,LISTENER_EXTERN://127.0.0.1:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERN:PLAINTEXT,LISTENER_EXTERN:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_EXTERN

      KAFKA_BROKER_ID: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I have two LISTENER because there is a second container mydb that does not work with the PLAINTEXT://127.0.0.1:9092 listener.

In fact, using

      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092

works with kafkacat. But then the other container can`t connect to kafka.

Now I want to connect to kafka via

kafkacat -b 127.0.0.1:9092 -t mytopic -L

but it gives this error

%6|1605191095.939|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 
127.0.0.1:9092/bootstrap: 
Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration 
(connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 2ms in state APIVERSION_QUERY)

the debug log for the broker is:

%7|1605191334.678|BROKER|rdkafka#producer-1| [thrd:app]: 127.0.0.1:9092/bootstrap: Added new broker with NodeId -1
%7|1605191334.678|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1605191334.678|CONNECT|rdkafka#producer-1| [thrd:app]: 127.0.0.1:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1605191334.678|BRKMAIN|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Enter main broker thread
%7|1605191334.678|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.5.0 (0x10500ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG OSXLD LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x2)
%7|1605191334.678|CONNECT|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Received CONNECT op
%7|1605191334.678|STATE|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1605191334.678|CONNECT|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1605191334.678|STATE|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1605191334.678|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: leader query
%7|1605191334.678|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1605191334.678|CONNECT|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connecting to ipv4#127.0.0.1:9092 (plaintext) with socket 7
%7|1605191334.679|CONNECT|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connected to ipv4#127.0.0.1:9092
%7|1605191334.679|CONNECTED|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Connected (#1)
%7|1605191334.679|FEATURE|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1605191334.679|STATE|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1605191334.679|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1605191334.681|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9092/bootstrap]: 127.0.0.1:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 2ms in state APIVERSION_QUERY) (_TRANSPORT)

What am I doing wrong?

Solution

I played around with your particular example, and couldn't get it to work.

For what it's worth, this Docker Compose is how I run Kafka on Docker locally, and it's accessible both from the host machine, and other containers.

You might find this blog useful if you want to continue with your existing approach and debug it further.