I have issue with topics created using AdminClient createTopics
.
In my application I have following sequence:
AdminClient.createTopics
AdminClient.createTopics
KafkaFuture
resultMy code is as follows:
adminClient
.createTopics(Collections.singleton(new NewTopic(targetTopic, 1, (short) 1)))
.values()
.get(targetTopic)
.get();
producer.send(new ProducerRecord<>(targetTopic, data));
From time to time producer doesn't see the created topic and throws following exception:
[Producer clientId=producer-1] Error while fetching metadata with correlation id 5 : {targetTopic=UNKNOWN_TOPIC_OR_PARTITION}
[Producer clientId=producer-1] Received unknown topic or partition error in produce request on partition targetTopic. The topic/partition may not exist or the user may not have Describe access to it
This issue is very rare (< 0,1% all created topics).
Is it guaranteed that when AdminClient.createTopics
Kafka future is completed then topic is created and Kafka producer should see that topic ?
If not then which method of topic creation can give me such guarantee ?
I am using kafka-clients:2.0.0 and Kafka HD service on Azure. My cluster consists of 3 Zookeeper and 3 Kafka nodes.
Kafka client is not guarenteed about atomic topic creation. You can read it from this documentation. When you create a topic, kafka will notify distributed systems about topics information so the action is running on different machines and these can't be atomic without distributed transactional management.