javaspring-bootspring-batch

Want to implement exponentialbackoff for a transaction in Spring Batch


Basically the title. I am using a JDBC item reader and JDBC item writer and I'm changing a particular status through the processor by using an API, if the API fails to change the status, I want to use exponential backoff to retry this at a later instant. I'm not able to figure out how to implement this


Solution

  • You have two options:

    1. Handle the retry operation manually in your item processor

    There are basically two ways to do that, either programmatically or in a declarative way.

    1.1 Programmatic approach

    You first define your retry template with a backoff policy as needed:

    @Bean
    public RetryTemplate retryTemplate() {
        // configure backoff policy
        ExponentialBackOffPolicy exponentialBackOffPolicy = new ExponentialBackOffPolicy();
        exponentialBackOffPolicy.setInitialInterval(1000);
        exponentialBackOffPolicy.setMultiplier(2.0);
        exponentialBackOffPolicy.setMaxInterval(10000);
    
        // configure retry policy
        SimpleRetryPolicy simpleRetryPolicy = new SimpleRetryPolicy();
        simpleRetryPolicy.setMaxAttempts(5);
    
        // configure retry template
        RetryTemplate retryTemplate = new RetryTemplate();
        retryTemplate.setBackOffPolicy(exponentialBackOffPolicy);
        retryTemplate.setRetryPolicy(simpleRetryPolicy);
    
        return retryTemplate;
    }
    

    Then use that retry template in your item processor:

    import org.springframework.batch.item.ItemProcessor;
    import org.springframework.retry.RetryCallback;
    import org.springframework.retry.RetryContext;
    import org.springframework.retry.support.RetryTemplate;
    
    public class MyRetryableItemProcessor implements ItemProcessor {
    
        RetryTemplate retryTemplate;
    
        public MyRetryableItemProcessor(RetryTemplate retryTemplate) {
            this.retryTemplate = retryTemplate;
        }
    
        @Override
        public Object process(Object item) throws Exception {
            return retryTemplate.execute(new RetryCallback<Object, Exception>() {
                @Override
                public Object doWithRetry(RetryContext retryContext) throws Exception {
                    // API call
                    return item;
                }
            });
        }
    }
    
    1.2 Declarative approach using annotations

    Here is an example:

    import org.springframework.batch.item.ItemProcessor;
    import org.springframework.retry.RetryCallback;
    import org.springframework.retry.RetryContext;
    import org.springframework.retry.annotation.Backoff;
    import org.springframework.retry.annotation.Retryable;
    import org.springframework.retry.support.RetryTemplate;
    import org.springframework.stereotype.Component;
    
    @Component
    public class MyAnnotationBasedRetryableItemProcessor implements ItemProcessor {
    
        @Override
        @Retryable(backoff = @Backoff(delay = 1000L, maxDelay = 10000, multiplier = 2.0))
        public Object process(Object item) throws Exception {
            // Do API call
            return item;
        }
    
    }
    

    2. Let Spring Batch handle the retry for you by using a fault-tolerant step

    In this case, you can set a custom RetryPolicy in your fault-tolerant step:

    @Bean
    public Step step(StepBuilderFactory stepBuilderFactory) {
        // configure backoff policy
        ExponentialBackOffPolicy exponentialBackOffPolicy = new ExponentialBackOffPolicy();
        exponentialBackOffPolicy.setInitialInterval(1000);
        exponentialBackOffPolicy.setMultiplier(2.0);
        exponentialBackOffPolicy.setMaxInterval(10000);
    
        // configure retry policy
        SimpleRetryPolicy simpleRetryPolicy = new SimpleRetryPolicy();
        simpleRetryPolicy.setMaxAttempts(5);
        
        return stepBuilderFactory.get("myStep")
                .<Integer, Integer>chunk(5)
                .reader(itemReader())
                .processor(itemProcessor())
                .writer(itemWriter())
                .faultTolerant()
                .retryPolicy(simpleRetryPolicy)
                .backOffPolicy(exponentialBackOffPolicy)
                .build();
    }
    

    Note that in this case, whenever your processor throws an exception for an item, the entire chunk is retried item by item (and each item will be re-processed in its own transaction).


    The examples above use spring-retry since you mentioned you have a preference for that. But the same ideas can be applied with any other fault-tolerance library.