javaubuntuapache-kafkanetworkx

Connection error to Kafka broker when publishing a message through java app

I'm trying to publish a kafka message running a java app (on my windows laptop) on an Kafka broker that's running on Ubuntu (on my windows laptop). The error I get is:

    [kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node -1 disconnected.
    [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
    [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

Here is the java code: public static void main(String[] args) {

        //create producer properties
        Properties properties = new Properties();

        //connect to localhost
        properties.setProperty("bootstrap.servers", "127.0.0.1:9092");

        //set producer properties
        properties.setProperty("key.serializer", StringSerializer.class.getName());
        properties.setProperty("value.serializer", StringSerializer.class.getName());

        //create the producer
        KafkaProducer<String, String> producer = new KafkaProducer<>(properties);
        //create a Producer Record
        ProducerRecord<String, String> producerRecord = new ProducerRecord<>("demo_java", "hello world");

        //send data
        producer.send(producerRecord);

        //tell the producer to send all data and block until done -- synchronous
        producer.flush();

        //flush and close the producer
        producer.close();
    }

What I've tried:

Running any of the kafka command (for example, kafka-topics.sh --bootstrap-server localhost:9092 --topic --list) runs successfully on the ubuntu server. I installed Ubuntu on my windows laptop.

Solution

  • It looks like the issue lies in how your Java app and the Kafka broker communicate due to network configurations. The Kafka broker's configuration is likely bound to localhost (127.0.0.1), which prevents external connections (even from the Windows-hosted Java app). Let’s debug and resolve it step by step.

    Steps to Resolve 1. Update Kafka Broker Configuration

    By default, Kafka listens only on localhost, meaning it’s only accessible from the same machine/environment. Since the broker is running on Ubuntu (inside WSL or a virtual machine), you need to configure Kafka to listen on an IP address that your Windows-hosted Java app can access.

    Locate the Kafka broker configuration file, typically found at:

    /path/to/kafka/config/server.properties

    Update the following settings:

    Advertised Listeners Add your Ubuntu’s network IP address so it’s accessible externally:

    advertised.listeners=PLAINTEXT://<ubuntu-ip-address>:9092

    Replace with the IP address of your Ubuntu machine. (You can get this by running ifconfig or ip a in your Ubuntu terminal.)

    For example:

    advertised.listeners=PLAINTEXT://192.168.1.10:9092

    Listeners Ensure Kafka is listening on all network interfaces:

    listeners=PLAINTEXT://0.0.0.0:9092

    After updating, restart your Kafka broker:

    ./bin/kafka-server-stop.sh
./bin/kafka-server-start.sh -daemon /path/to/config/server.properties

    2. Test Connectivity

    Ensure that your Windows machine can communicate with the Kafka broker.

    From Windows, run:

    telnet <ubuntu-ip-address> 9092

    If successful, this confirms that the Kafka port is open and accessible.

    Verify Kafka’s listener using the following commands on Ubuntu:

    netstat -tuln | grep 9092

    You should see Kafka bound to 0.0.0.0:9092.

    3. Update Your Java App's bootstrap.servers

    Change the bootstrap.servers property in your Java code to point to the Ubuntu machine instead of 127.0.0.1. For example:

    properties.setProperty("bootstrap.servers", "192.168.1.10:9092");

    Replace 192.168.1.10 with the actual IP address of your Ubuntu machine.

    4. Confirm Kafka Topic Exists

    Make sure the topic (demo_java) exists in your Kafka instance. Run the following command on the Ubuntu Kafka machine:

    kafka-topics.sh --bootstrap-server 192.168.1.10:9092 --list

    If the topic doesn’t exist, create it:

    kafka-topics.sh --bootstrap-server 192.168.1.10:9092 --create --topic demo_java --partitions 1 --replication-factor 1

    5. Verify Firewall Rules

    Since you’re running Kafka across different environments, ensure the following:

    The Ubuntu firewall (if enabled) allows traffic on port 9092. You can check

    and add a rule:
sudo ufw allow 9092
sudo ufw reload

    The Kafka port 9092 is open for incoming traffic on both Ubuntu and Windows firewalls.

    6. Re-run Your Java App

    Run the Java app again. If everything is set up correctly, the producer should successfully connect to the Kafka broker and publish the message.

    Troubleshooting

    If the connection still fails: Double-check Kafka logs (logs/server.log) on the Ubuntu machine for binding or network errors.

    If localhost is hardcoded anywhere: Make sure all configurations use the Ubuntu machine's IP address instead of localhost.