apache-kafkaretention

changing kafka retention period during runtime


With Kafka 0.8.1.1, how do I change the log retention time while it's running? The documentation says the property is log.retention.hours, but trying to change it using kafka-topics.sh returns this error

$ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config topic.log.retention.hours=24
Error while executing topic command requirement failed: Unknown configuration "topic.log.retention.hours".
java.lang.IllegalArgumentException: requirement failed: Unknown configuration "topic.log.retention.hours".
    at scala.Predef$.require(Predef.scala:145)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:138)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:137)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at scala.collection.JavaConversions$JEnumerationWrapper.foreach(JavaConversions.scala:479)
    at kafka.log.LogConfig$.validateNames(LogConfig.scala:137)
    at kafka.log.LogConfig$.validate(LogConfig.scala:145)
    at kafka.admin.TopicCommand$.parseTopicConfigsToBeAdded(TopicCommand.scala:171)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:95)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:93)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
    at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:93)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

Solution

  • log.retention.hours is a property of a broker which is used as a default value when a topic is created. When you change configurations of currently running topic using kafka-topics.sh, you should specify a topic-level property.

    A topic-level property for log retention time is retention.ms.

    From Topic-level configuration in Kafka 0.8.1 documentation:

    So the correct command depends on the version. Up to 0.8.2 (although docs still show its use up to 0.10.1) use kafka-topics.sh --alter and after 0.10.2 (or perhaps from 0.9.0 going forward) use kafka-configs.sh --alter

    $ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config retention.ms=86400000
     
    

    You can check whether the configuration is properly applied with the following command.

    $ bin/kafka-topics.sh --describe --zookeeper zk.yoursite.com --topic as-access
    

    Then you will see something like below.

    Topic:as-access  PartitionCount:3  ReplicationFactor:3  Configs:retention.ms=86400000