apache-kafkaretention

set config retention.ms=3600000 still data not delete from Kafka

I have set the retention.ms=3600000 by below command but still there is lots of data on disk after 1 hour. My disk got full due to huge data coming to Kafka.

./bin/kafka-topics.sh --zookeeper zookeeper:2181 --alter --topic topic_1 --config retention.ms=3600000

Describe command

 ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --describe --topics-with-overrides
    Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:3     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic:topic_1    PartitionCount:3        ReplicationFactor:3     Configs:retention.ms=3600000
    Topic:topic_2    PartitionCount:3        ReplicationFactor:3     Configs:retention.ms=3600000
    Topic:topic_3    PartitionCount:3        ReplicationFactor:3     Configs:retention.ms=3600000,retention.bytes=104857600

Can anyone give advice why kafka not delete the data after 1 hours.?

Solution

  • From the describe command result, topic retention policy is set to compact which will enable log compaction instead of deleting and will keep the latest data for each key. To delete all the data older than the retention period, you need to set retention policy to delete.

    ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --alter --topic topic_1 --config cleanup.policy=delete