pythonkafka-python

When kafka consumer connected to a broker by domain name, if the ip changed after broker crash, will the kafka consumer reconnect?


My kafka consumer is connected to a cluster of brokers. There is a domain name server in between. Each broker has a domain name associated with an IP address. Problem is, the IPs are not static and for some reason, I have to restart the broker. The consumers are configured to reconnect. My question is when it will try to reconnect, will the consumer resolve the domain name to the new IP address or it will use the previously resolved IP address?


Solution

  • Domain name resolution is performed by the underlying operating system's DNS resolver, which caches the resolved IP address for a certain period of time (known as the "time to live" or TTL). When your Kafka consumer is configured to reconnect to a Kafka cluster after a broker failure, it will attempt to resolve the domain name associated with the broker to an IP address every time it attempts to establish a new connection.

    If you restart a broker and its IP address changes, the consumer will eventually attempt to connect to the new IP address once the TTL for the domain name has expired and a new lookup is performed. A similar issue has been fixed on the latest release of Kafka

    GitHub Issue: https://github.com/akka/alpakka-kafka/issues/734

    Kafka Ticket: https://issues.apache.org/jira/browse/KAFKA-7755

    But it's a good idea to monitor your Kafka consumer's logs and behavior after a broker restart to ensure that it is successfully reconnecting to the cluster.