Consumer Behaviour
Upon start-up, my consumer logs shows many lines of
Disconnecting from node X due to socket connection setup timeout. The timeout value is XX ms.
and does not seem to consume. Weirdly, it has consumed records successfully before, but most of the time it does not.
Producer Behaviour
The producer produces fine for about a minute, then starts throwing the following errors
ERROR org.springframework.kafka.support.LoggingProducerListener - Exception thrown when sending a message with key=XX and payload=XX to topic XX and partition X;
org.apache.kafka.common.errors.TimeoutException: Expiring X record(s) for topic-name:12000 ms has passed since batch creation
What could be the error given that the Kafka Cluster is healthy and working fine, and that producing and consuming seems to be working intermittently?
This took embarrassingly long to solve. The error was with something I ruled out quite early on because I didn't check thoroughly enough, and that is connectivity with Kafka cluster. Since the producing/consuming seemed to have worked partially, and I checked that connectivity to one Kafka broker did work, I didn't think it would be a firewall issue.
Alas, it was. When I checked the connection to all the Kafka brokers, I realised that some could not go through, and after the firewall was punched, the Kafka client started behaving as expected.
Why the set-up was only done halfway is unclear, but the resolution here was to make sure to check the connection to all Kafka brokers and not just one!