apache-kafkakafka-cluster

Consumen multiple network interfaces of single machine for kafka cluster


I have a Linux machine with 3 network interfaces, let's say IPs are 192.168.1.101,192.168.1.102,192.168.1.103 I want to consume all 3 IPs of this single node to create a Kafka cluster with other nodes, Should all 3 IPs have their separate brokers? Also using nic bonding is not recommended, all IPs need to be utilized


Solution

  • Overall, I'm not sure why you'd want to do this... If you are using separate volumes (log.dirs) for each address, then maybe you'd want separate Java processes, sure, but you'd still be sharing the same memory, and having that machine be a single point of failure.

    In any case, you can set one process to have advertised.listeners list out each of those addresses for clients to communicate with, however, you'd still have to deal with port allocations in the OS, so you might need to set listeners like so

    listeners=PLAINTEXT_1://0.0.0.0:9092,PLAINTEXT_2://0.0.0.0:9093,PLAINTEXT_3://0.0.0.0:9094
    

    And make sure you have listener.security.protocol.map setup as well using those names

    Note that clients will only communicate with the leader topic-partition at any time, so if you have one broker JVM process and 3 addresses for it, then really only one address is going to be utilized. One optimization for that could be your intra-cluster replication can use a separate NIC.