javadockerapache-kafka

Connect to Kafka running in Docker

I setup a single node Kafka Docker container on my local machine like it is described in the Confluent documentation (steps 2-3).

In addition, I also exposed Zookeeper's port 2181 and Kafka's port 9092 so that I'll be able to connect to them from a client running on local machine:

$ docker run -d \
    -p 2181:2181 \
    --net=confluent \
    --name=zookeeper \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    confluentinc/cp-zookeeper:4.1.0

$ docker run -d \
    --net=confluent \
    --name=kafka \
    -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka:4.1.0

Problem: When I try to connect to Kafka from the host machine, the connection fails because it can't resolve address: kafka:9092.

Here is my Java code:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("client.id", "KafkaExampleProducer");
props.put("key.serializer", LongSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
KafkaProducer<Long, String> producer = new KafkaProducer<>(props);
ProducerRecord<Long, String> record = new ProducerRecord<>("foo", 1L, "Test 1");
producer.send(record).get();
producer.flush();

The exception:

java.io.IOException: Can't resolve address: kafka:9092
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.common.network.Selector.connect(Selector.java:214) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:265) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:266) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:176) [kafka-clients-2.0.0.jar:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.nio.channels.UnresolvedAddressException: null
    at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_144]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_144]
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:na]
    ... 7 common frames omitted

Question: How to connect to Kafka running in Docker? My code is running from host machine, not Docker.

Note: I know that I could theoretically play around with DNS setup and /etc/hosts but it is a workaround - it shouldn't be like that.

There is also similar question here, however it is based on ches/kafka image. I use confluentinc based image which is not the same.

Solution

  • tl;dr - A simple port forward from the container to the host will not work... Hosts files (e.g. /etc/hosts on *NIX systems) should not be modified to work around Kafka networking, as this solution is not portable.

    1) What exact IP/hostname + port do you want to connect to? Make sure that value is set as advertised.listeners (not advertised.host.name and advertised.port, as these are deprecated) on the broker. If you see an error such as Connection to node -1 (localhost/127.0.0.1:9092), then that means your app container tries to connect to itself. Is your app container also running a Kafka broker process? Probably not.

    2) Make sure that the server(s) listed as part of bootstrap.servers are actually resolvable. E.g ping an IP/hostname, use netcat to check ports... If your clients are in a container, you need to do this from the container, not (only) your host. Use docker exec if the container isn't immediately crashing to get to its shell.

    3) If running a process from the host, rather than another container, to verify the ports are mapped correctly on the host, ensure that docker ps shows the kafka container is mapped from 0.0.0.0:<host_port> -> <advertised_listener_port>/tcp. The ports must match if trying to run a client from outside the Docker network. You do not need port forwarding between two containers; use links / docker networks

    The below answer uses confluentinc docker images to address the question that was asked, not wurstmeister/kafka. If you have KAFKA_ADVERTISED_HOST_NAME variable set, remove it (it's a deprecated property). The same variables apply to apache/kafka image.

    The following sections try to aggregate all the details needed to use another image. For other, commonly used Kafka images, it's all the same Apache Kafka running in a container.
    You're just dependent on how it is configured. And which variables make it so.

    apache/kafka

    There's now an official image! Find links below for example Compose files.

    wurstmeister/kafka

    As of Oct 2023, this no longer exists in DockerHub. Wasn't maintained past 2022, anyway.

    Refer their README section on listener configuration, Also read their Connectivity wiki.

    bitnami/kafka

    If you want a small container, try these. The images are much smaller than the Confluent ones and are much more well maintained than wurstmeister. Refer their README for listener configuration.

    debezium/kafka

    Docs on it are mentioned here.

    Note: advertised host and port settings are deprecated. Advertised listeners covers both. Similar to the Confluent containers, Debezium can use KAFKA_ prefixed broker settings to update its properties.

    Others

    • ubuntu/kafka requires you to add --override advertised.listeners=kafka:9092 via Docker image args... I find that less portable than environment variables, so not recommended
    • spotify/kafka is deprecated and outdated.
    • fast-data-dev or lensesio/box are great for an all in one solution, with Schema Registry, Kafka Connect, etc, but are bloated if you only want Kafka. Plus, it's a Docker anti pattern to run many services in one container
    • Your own Dockerfile - Why? Is something incomplete with these others? Start with a pull request, not starting from scratch.

    For supplemental reading, a fully-functional docker-compose, and network diagrams, see this blog by @rmoff

    Answer

    The Confluent quickstart (Docker) document assumes all produce and consume requests will be within the Docker network.

    You could fix the problem of connecting to kafka:9092 by running your Kafka client code within its own container as that uses the Docker network bridge, but otherwise you'll need to add some more environment variables for exposing the container externally, while still having it work within the Docker network.

    First add a protocol mapping of PLAINTEXT_HOST:PLAINTEXT that will map the listener protocol to a Kafka protocol

    Key: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
    Value: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

    Then setup two advertised listeners on different ports. (kafka here refers to the docker container name; it might also be named broker, so double check your service + hostnames).

    Key: KAFKA_ADVERTISED_LISTENERS
    Value: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

    Notice the protocols here match the left-side values of the protocol mapping setting above

    When running the container, add -p 29092:29092 for the host port mapping, and advertised PLAINTEXT_HOST listener.

    So... (with the above settings)

    If something still doesn't work, KAFKA_LISTENERS can be set to include <PROTOCOL>://0.0.0.0:<PORT> where both options match the advertised setting and Docker-forwarded port

    Client on same machine, not in a container

    Advertising localhost and the associated port will let you connect outside of the container, as you'd expect.

    In other words, when running any Kafka Client outside the Docker network (including CLI tools you might have installed locally), use localhost:29092 for bootstrap servers (requires Docker port forwarding)

    Client on another machine

    If trying to connect from an external server, you'll need to advertise the external hostname/ip (e.g. 192.168.x.y) of the host as well as/in place of localhost.
    Simply advertising localhost with a port forward will not work because Kafka protocol will still continue to advertise the listeners you've configured.

    This setup requires Docker port forwarding and router port forwarding (and firewall / security group changes) if not in the same local network, for example, your container is running in the cloud and you want to interact with it from your local machine.

    Client (or another broker) in a container, on the same host

    This is the least error-prone configuration; you can use DNS service names directly.

    When running an app in the Docker network, use Docker service names such as kafka:9092 (see advertised PLAINTEXT listener config above) for bootstrap servers, just like any other Docker service communication (doesn't require any port forwarding)

    If you use separate docker run commands, or Compose files, you need to define a shared network manually using compose networks section or docker network --create

    See the example Compose file for the full Confluent stack using Kraft or more minimal one (with Zookeeper) for a single broker.

    For apache/kafka image, there's example files in the Kafka Github repo.

    If using multiple brokers, then they need to use unique hostnames + advertised listeners. See example

    Related question

    Connect to Kafka on host from Docker (ksqlDB)

    Appendix

    For anyone interested in Kubernetes deployments: