spring-mvcapache-kafkaibm-eventstreams

Configure kafka topic retention policy during creation in spring-mvc?


Configure retention policy of all topics during creation

Trying to configure rentention.ms using spring, as I get an error of:

Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.PolicyViolationException: Invalid retention.ms specified. The allowed range is [3600000..2592000000] From what I've read the new value is -1 (infinity) so is out of that range

Following what was in How to configure kafka topic retention policy during creation in spring-mvc? ,I added the below code but it seems to have no effect. Any ideas/hints on how might solve this?

ApplicationConfigurationTest.java
  @test
   public void kafkaAdmin () {
       KafkaAdmin admin = configuration.admin();
       assertThat(admin, instanceOf(KafkaAdmin.class));
   }

ApplicationConfiguration.java
    @Bean
   public KafkaAdmin admin() {
       Map<String, Object> configs = new HashMap<>();
       configs.put(TopicConfig.RETENTION_MS_CONFIG, "1680000");
       return new KafkaAdmin(configs);
   }

Solution

  • Found the solution by setting the value
    spring.kafka.streams.topic.retention.ms: 86400000
    in application.yml.
    Our application uses spring mvc, hence the spring notation.
    topic.retention.ms is the value that needs to be set in the streams config
    86400000 is a random value just used as it is in range of [3600000..2592000000]