Hi I am writing a service in Go and Kafka and I need to implement a delete all endpoint which would delete all records from a specific topic. However I can not find a proper way to do that. I am using the Sarama library for Kafka.
So far the only two ways I can find to implement delete all is by deleting the topic which does not seem to be an efficient way to handle this problem and the second one is using the DeleteRecords function from the Sarama library, however this function Deletes records whose offset is smaller than the given offset of the corresponding partition. Which means that I have to get the latest offset first.
Basically I am looking for the best way to do such a thing. Could anyone help me? What are the best practices? Maybe I have missed something. I would really appreciate some examples. Thank you!
If you want to prune all the messages, another way to do that is to reduce the retention of the topic to a small value (e.g. 100ms). Wait for the brokers to remove all the records from the topic and then set the topic retention to its original value. Here’s how to do it.
First, set the retention time to 100 milliseconds.
kafka-configs --zookeeper localhost:2181 \
--entity-type topics \
--entity-name my-topic \
--alter --add-config retention.ms=100
Alternative Solution:
Delete a topic and create it again Not as elegant as the previous two approaches, yet it might be an easier solution in some cases (e.g. if topic creation is scripted).
kafka-topics --bootstrap-server localhost:9092 \
--topic my-topic \
--delete
Then create it again:
kafka-topics --bootstrap-server localhost:9092 \
--topic my-topic \
--create \
--partitions <number_of_partitions> \
--replication-factor <replication_factor>