springspring-bootspring-batch

Load Multiple CSV files into database using Spring Batch


I want to load multiple CSV files into mysql database at single table using Spring Batch. The path of the files are derived from the following method.

    public List<String> getFilePath() {
    String inputPath = "E:\\input";
    List<String> inputCSVPaths = new ArrayList<String>();
    Map<String, List<String>> inputInfo = new HashMap<String, List<String>>();

    File inputFolder = new File(inputPath);
    File[] inputFiles = inputFolder.listFiles();
    for (File file : inputFiles) {
        inputCSVPaths.add(file.getAbsolutePath());
    }
    inputInfo.put("Introduction", inputCSVPaths);
    List<String> inputFile = inputInfo.get("Introduction");
    System.out.println("Input File :"+inputFile);
    return inputFile;

}

There are total 3 CSV files. But it reads only onle file and inserts data of only that CSV file. Is there wrong in getting resources.

@Autowired
private FilePathDemo filePathDemo;

 @Bean
public MultiResourceItemReader<Introduction> multiResourceItemReader() throws IOException {

    MultiResourceItemReader<Introduction> multiReader = new MultiResourceItemReader<Introduction>();
    ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();

    Resource[] resources;
    String filePath = "file:";
    List<String> path = filePathDemo.getFilePath();
    for (String introPath : path) {
        System.out.println("File Path of the Introduction CSV :" + introPath);

        resources = patternResolver.getResources(filePath + introPath);
        multiReader.setResources(resources);
    }

        FlatFileItemReader<Introduction> flatReader = new FlatFileItemReader<Introduction>();
    multiReader.setDelegate(flatReader);
    flatReader.setLinesToSkip(1);
    flatReader.setLineMapper(new DefaultLineMapper<Introduction>() {
        {
            setLineTokenizer(new DelimitedLineTokenizer() {
                {
                    setNames(new String[] { "id", "name", "age", "phoneNo"});
                }
            });
            setFieldSetMapper(new BeanWrapperFieldSetMapper<Introduction>() {
                {
                    setTargetType(Introduction.class);
                }
            });
        }
    });

    flatReader.close();
    multiReader.close();

    return multiReader;
}

Solution

  • There are two issues with your configuration:

    You can find an example of how to configure the MultiResourceItemReader here: https://docs.spring.io/spring-batch/4.0.x/reference/html/readersAndWriters.html#multiFileInput