apache-kafkaretentionkafka-topic

Does updating broker settings (topic defaults) change already existing topics?


In kafka there is a log.retention.ms configuration property that can be set at the Broker level and it can be overridden by retention.ms on topic level.

Imagine I have a topic in my Kafka cluster with default config - no explicitly set retention. When I created the topic, the Broker level log.retention.ms was set to 7 days. In the meantime I changed this setting to 30 days. Does this mean that my existing topic will use the configuration that existed when the topic was created (7 days) or does it now use the new Broker config of 30 days?

Since the not-explicitly-set config values at topic level are not shown when I do kafka-topics.sh --describe --topic foo I find it difficult to verify.


Solution

  • Does this mean that my existing topic will use the configuration that existed when the topic was created (7 days) or does it now use the new Broker config of 30 days?

    Your topic will use the configuration settings when it was created, that is, 7 days.

    Only newly created topics will be set with a retention of 30 days. Note that setting this at the broker level implies a "default" value for topics that don't specify the retention time, but modifying this at the broker level won't affect existing topics, there's no "on cascade" update mechanism here.

    Short answer: No. The only action that can alter the retention time set for a topic when it was created is specifically changing this value, for example, by using kafka-topics.sh --alter command.