[SOLVED] Kafka producer says "unknown_topic_or

Kafka producer says "unknown_topic_or_partition"

I've been trying to make kafka-docker work for a few days now and I don't know what I'm doing wrong. Right now, I can't access any topic with my ruby-kafka client because the node "doesn't exist". This is my docker-compose.yml file:

version: '2'
services:
  zookeeper:
   image: wurstmeister/zookeeper
   ports:
     - "2181:2181"
  kafka:
    image: wurstmeister/kafka:0.9.0.1
    ports:
      - "9092:9092"
    links:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  kafka2:
    image: wurstmeister/kafka:0.9.0.1
    ports:
      - "9093:9092"
    links:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  kafka3:
    image: wurstmeister/kafka:0.9.0.1
    ports:
      - "9094:9092"
    links:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I specify "KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'" because I want to create topics by hand, so I entered into my first broker container and typed this:

./kafka-topics.sh --create --zookeeper 172.19.0.2:2181 --topic test1 --partitions 4 --replication-factor 3

And everything seems fine:

./kafka-topics.sh --list --zookeeper 172.19.0.2:2181 -> test1

But, when I try to do this:

./kafka-console-producer.sh --broker-list localhost:9092 --topic test1

It says:

WARN Error while fetching metadata with correlation id 24 : {test1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

If I create again the topic, it says it already exists, so I don't know what is happening anymore.

Solution

You need to get your networking configuration right, as Kafka works across hosts and needs to be able to access them all.

My post explains it in detail.

You might also want to reference https://github.com/confluentinc/cp-docker-images/blob/5.0.0-post/examples/cp-all-in-one/docker-compose.yml for an example of a working Docker Compose.