Kafka docker compose external connection

I want to expost 9093 to outside of the docker container. When I set the kafka-0 ports exposed to 9093 and the KAFKA_ADVERTISED_LISTENERS as follow, i am unable to connect to localhost:9093 as shown in the following docker-compose file.

version: '3'

services:
  kafka-0:
    image: confluentinc/cp-kafka:5.2.1
    container_name: kafka-0
    hostname: kafka-0
    ports:
      - "9093:9092"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=wise-nlp-zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:5.3.1
    container_name: zookeeper
    ports:
      - "2182:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181

However, when i change to

ports:
 - "9092:9092"

and

- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9092

I am able to connect to kafka broker localhost:9092.

How can i change external port to 9093 for applications to connect? I want to set up multiple brokers.

Solution

Why's it not working currently?

Advertised listener(s) (as defined in KAFKA_ADVERTISED_LISTENERS) are the host and port that the broker returns to the client in its initial connection for it to use in subsequent connections.

If you want external clients to use 9093 then KAFKA_ADVERTISED_LISTENERS=…PLAINTEXT_HOST://localhost:9093 is correct. However, you've not configured your KAFKA_LISTENERS, which if you check the broker log when it starts up will default to the value set by KAFKA_ADVERTISED_LISTENERS:

kafka-0      |  listeners = PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093

So in this state, the broker is listening on port 9093, but with this Docker Compose instruction you've redirected external connections into the container on 9093 to 9092 within the container:

    ports: 
      - "9093:9092"

➜ docker ps
CONTAINER ID        IMAGE                             … PORTS                                        NAMES
8b934ef4145c        confluentinc/cp-kafka:5.4.1       … 0.0.0.0:9093->9092/tcp                       kafka-0

So your external connections will go to port 9092 in the container—and the broker is not listening on this port. You can verify this with nc:

-- Port 9093 is open on the host machine
➜ nc -vz localhost 9093
Connection to localhost port 9093 [tcp/*] succeeded!

-- Port 9092 is _not_ open on the Kafka container
➜ docker exec -it kafka-0 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused

❌ You'll see that a client connection fails

➜ kafkacat -b localhost:9093 -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure

How can you fix it?

You can either:

Change the listener to be on the port that you target with the Docker port redirect. This will work but personally I think is more confusing.
Change the Docker port redirect to target the port on which the listener is on. This is the option I would use as it is clearer (e.g. port 9093 is used throughout, rather than mixing 9092 and 9093 together)

Option 1: Change the listener to be on the port that you target with the Docker port redirect

version: '3'

services:
  kafka-0:
    image: confluentinc/cp-kafka:5.4.1
    container_name: kafka-0
    ports:
      - "9093:9092"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
      - KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9092
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.1
    container_name: zookeeper
    ports:
      - "2182:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181

✅Test:

➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
 1 brokers:
  broker 1 at localhost:9093 (controller)

Option 2: Change the Docker port redirect to target the port on which the listener is on

version: '3'

services:
  kafka-0:
    image: confluentinc/cp-kafka:5.4.1
    container_name: kafka-0
    ports:
      - "9093:9093"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
      # If you don't specify KAFKA_LISTENERS it will default to the ports used in
      # KAFKA_ADVERTISED_LISTENERS, but IMO it's better to be explicit about these settings
      - KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.1
    container_name: zookeeper
    ports:
      - "2182:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181

✅Test

➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
 1 brokers:
  broker 1 at localhost:9093 (controller)

Connecting to Kafka from within the Docker network

The examples above are about connecting to Kafka from the Docker host. If you want to connect to it from within the Docker network (e.g. another container) you need to use kafka-0:29094 as the broker host and IP. If you try to use localhost:9093 then the client container will resolve localhost to its own container, and thus fail.

Multiple brokers

See here for an example Docker Compose with multiple Kafka brokers.