springspring-batch

Spring batch : FlatFileItemWriter header never called


I have a weird issue with my FlatFileItemWriter callbacks. I have a custom ItemWriter implementing both FlatFileFooterCallback and FlatFileHeaderCallback. Consequently, I set header and footer callbacks in my FlatFileItemWriter like this :

ItemWriter Bean

@Bean
@StepScope
public ItemWriter<CityItem> writer(FlatFileItemWriter<CityProcessed> flatWriter, @Value("#{jobExecutionContext[inputFile]}") String inputFile) {
        CityItemWriter itemWriter = new CityItemWriter();
        flatWriter.setHeaderCallback(itemWriter);
        flatWriter.setFooterCallback(itemWriter);
        itemWriter.setDelegate(flatWriter);
        itemWriter.setInputFileName(inputFile);
        return itemWriter;
}

FlatFileItemWriter Bean

@Bean
@StepScope
public FlatFileItemWriter<CityProcessed> flatFileWriterArchive(@Value("#{jobExecutionContext[outputFileArchive]}") String outputFile) {
    FlatFileItemWriter<CityProcessed> flatWriter = new FlatFileItemWriter<CityProcessed>();
    FileSystemResource isr;
    isr = new FileSystemResource(new File(outputFile));
    flatWriter.setResource(isr);
    DelimitedLineAggregator<CityProcessed> aggregator = new DelimitedLineAggregator<CityProcessed>();
    aggregator.setDelimiter(";");
    BeanWrapperFieldExtractor<CityProcessed> beanWrapper = new BeanWrapperFieldExtractor<CityProcessed>();
    beanWrapper.setNames(new String[]{
        "country", "name", "population", "popUnder25", "pop25To50", "pop50to75", "popMoreThan75"
    });
    aggregator.setFieldExtractor(beanWrapper);
    flatWriter.setLineAggregator(aggregator);
    flatWriter.setEncoding("ISO-8859-1");
    return flatWriter;
}

Step Bean

@Bean
public Step stepImport(StepBuilderFactory stepBuilderFactory, ItemReader<CityFile> reader, ItemWriter<CityItem> writer, ItemProcessor<CityFile, CityItem> processor,
    @Qualifier("flatFileWriterArchive") FlatFileItemWriter<CityProcessed> flatFileWriterArchive, ExecutionContextPromotionListener executionContextListener) {
    return stepBuilderFactory.get("stepImport").<CityFile, CityItem> chunk(10).reader(reader(null)).processor(processor).writer(writer).stream(flatFileWriterArchive)
        .listener(executionContextListener).build();
}

I have the classic content in my writeFooter, writeHeader and write methods.

ItemWriter code

public class CityItemWriter implements ItemWriter<CityItem>, FlatFileFooterCallback, FlatFileHeaderCallback, ItemStream {
    private FlatFileItemWriter<CityProcessed> writer;
    private static int totalUnknown = 0;
    private static int totalSup10000 = 0;
    private static int totalInf10000 = 0;
    private String inputFileName = "-";

    public void setDelegate(FlatFileItemWriter<CityProcessed> delegate) {
        writer = delegate;
    }

    public void setInputFileName(String name) {
        inputFileName = name;
    }

    private Predicate<String> isNullValue() {
        return p -> p == null;
    }

    @Override
    public void write(List<? extends CityItem> cities) throws Exception {
        List<CityProcessed> citiesCSV = new ArrayList<>();
        for (CityItem item : cities) {
             String populationAsString = "";
             String less25AsString = "";
             String more25AsString = "";
            /*
             * Some processing to get total Unknown/Sup 10000/Inf 10000
             * and other data
             */
            // Write in CSV file
            CityProcessed cre = new CityProcessed();
            cre.setCountry(item.getCountry());
            cre.setName(item.getName());
            cre.setPopulation(populationAsString);
            cre.setLess25(less25AsString);
            cre.setMore25(more25AsString);
            citiesCSV.add(cre);
        }
        writer.write(citiesCSV);
    }

    @Override
    public void writeFooter(Writer fileWriter) throws IOException {
        String newLine = "\r\n";
        String totalUnknown= "Subtotal:;Unknown;" + String.valueOf(nbUnknown) + newLine;
        String totalSup10000 = ";Sum Sup 10000;" + String.valueOf(nbSup10000) + newLine;
        String totalInf10000 = ";Sum Inf 10000;" + String.valueOf(nbInf10000) + newLine;
        String total = "Total:;;" + String.valueOf(nbSup10000 + nbInf10000 + nbUnknown) + newLine;
        fileWriter.write(newLine);
        fileWriter.write(totalUnknown);
        fileWriter.write(totalSup10000);
        fileWriter.write(totalInf10000);
        fileWriter.write(total );
    }

    @Override
    public void writeHeader(Writer fileWriter) throws IOException {
        String newLine = "\r\n";
        String firstLine= "FILE PROCESSED ON: ;" + new SimpleDateFormat("MM/dd/yyyy").format(new Date()) + newLine;
        String secondLine= "Filename: ;" + inputFileName + newLine;
        String colNames= "Country;Name;Population...;...having less than 25;...having more than 25";
        fileWriter.write(firstLine);
        fileWriter.write(secondLine);
        fileWriter.write(newLine);
        fileWriter.write(colNames);
    }

    @Override
    public void close() throws ItemStreamException {
        writer.close();
    }

    @Override
    public void open(ExecutionContext context) throws ItemStreamException {
        writer.open(context);
    }

    @Override
    public void update(ExecutionContext context) throws ItemStreamException {
        writer.update(context);
    }
}

When I run my batch, I only have the data for each city (write method part) and the footer lines. If I comment the whole content of write method and footer callback, I still don't have the header lines. I tried to add a System.out.println() text in my header callback, it looks like it's never called.

Here is an example of the CSV file produced by my batch :

France;Paris;2240621;Unknown;Unknown
France;Toulouse;439553;Unknown;Unknown
Spain;Barcelona;1620943;Unknown;Unknown
Spain;Madrid;3207247;Unknown;Unknown
[...]
Subtotal:;Unknown;2
;Sum Sup 10000;81
;Sum Inf 10000;17
Total:;;100

What is weird is that my header used to work before, when I added both footer and header callbacks. I didn't change them, and I don't see what I've done in my code to "broke" my header callback... And of course, I have no save of my first code. Because I see only now that my header has disappeared (I checked my few last files, and it looks like my header is missing for some time but I didn't see it), I can't just remove my modifications to see when/why it happens.

Do you have any idea to solve this problem ?

Thanks


Solution

  • When using Java config as you are, it's best to return the most specific type possible (the opposite of what you're normally told to do in java programming). In this case, your writer is returning ItemWriter, but is step scoped. Because of this a proxy is created that can only see the type that your java config returns which in this case is ItemWriter and does not expose the methods on the ItemStream interface. If you return CityItemWriter, I'd expect things to work.