We are trying to publish data form db2-IIDR (IBM CDC) to Kafka on AWS . Subscription fails due to below error-
An error occurred during the conversation with Kafka.
Error: org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for cdckafka-subsname-commitstream-0
Are there any pre-requisites to be performed to publish the streams from IIDR to Kafka on aws ?
Some more details -
-Kafka cluster is running on AWS
-IIDR CDC engines --> on premise (both source and target)
-On premise IPs have been white-listed and I can ping/telnet ports from on-prem to aws and vice versa .
Thanks!
You need to set up your Kafka brokers with listeners that will work with external clients. You can see details here.
Simply pinging from client on-premises to AWS is not enough - you need to validate it with a Kafka client such as kafkacat