spring-batchbatch-processingspring-batch-taskletspring-batch-job-monitoring

Spring Batch step write_count less than read_count and filter and skip counts are all zero


We are facing a situation in Spring Batch that step write_count sometimes is less than read_count and filter_count and read_skip_count, process_skip_count and write_skip_count are all zero while the job has been completed successfully with no exception. We are still investigating but wished to know how can this happen and what can cause that?
Just to add we have 2 different writers for this step and we use classifier to switch between them under some conditions.
Here is the step configuration code:

    public Step step() {
    return stepBuilderFactory.get("jobName")
        .<InputTransaction, OutputTransaction>chunk(500)
        .reader(itemReader)
        .processor(itemProcessor)
        .writer(classifierCompositeItemWriter())
        .faultTolerant()
        .skipLimit(Integer.MAX_VALUE)
        .skip(RuntimeException.class)
        .listener(adjustTransactionItemWriter)
        .listener(adjustTransactionItemLogFileWriter)
        .build();
}

Solution

  • After investigation, we identified the root cause of the problem. It relates to how Spring Batch manages flushing changes of persistent entities to the database. The issue arises not during the read-process-write phase, but when the commit fails. At the end of read-process-write, data has not yet been flushed. The commit triggers the flush, which may raise an exception (e.g., OptimisticLockingFailureException). When the commit fails, the old values for the step execution are restored to their pre-chunk state (including writeCount). the exception dose not count as a writeCountError. The solution is to force a flush by calling .flush() on the entity manager at the end of the writer.

    Details about the issue is discussed here: https://github.com/spring-projects/spring-batch/issues/1826