xmloutputspring-batchpartitioner

Springbatch dynamic multiple xml File writer


I have to do a batch that :
read some data from DB (each row is an item, this is fine)
then do some process to add some more data (more data is always better ;) )
then here is my problem, I have to write each item in an xml file who's name depends of the item's data.

for exemple I have
ItemA (attr1=toto, attr2=foo, attr3=myNonKeyData...)=>goes in toto_foo.xml
ItemB (attr1=toto, attr2=foo, attr3=myNonKeyData...)=>goes in toto_foo.xml

ItemC (attr1=tata, attr2=foo...)=>goes in tata_foo.xml
...

I can't see how to do this with only one batch run once.
I have too much keys and possible output files to do a classifier.
Maybe using the partitioner could be a good idea even it seems not designed for that.


Solution

  • This is what I have understood.

    Create an composite ItemWriter(inspired by org.springframework.batch.item.support.CompositeItemWriter). Here is some pseudocode

    public class MyCompositeItemWriter<T> implements ItemStreamWriter<T> {
    
    // The String key is the file-name(toto_foo.xml, tata_foo.xml etc)
    //The map will be empty to start with as we do not know how many files will be created. 
    private Map<String, ItemWriter<? super T>> delegates;
    
    private boolean ignoreItemStream = false;
    
    public void setIgnoreItemStream(boolean ignoreItemStream) {
        this.ignoreItemStream = ignoreItemStream;
    }
    
    @Override
    public void write(List<? extends T> items) throws Exception {
    
        for(T item : items) {
            ItemWriter<? super T> writer = getItemWriterForItem(item);
            // Writing one item ata time might be inefficent. You can optimize this by grouping items by fileName. 
            writer.write(item);     
        }       
    }
    
    
    private getItemWriterForItem(T item) {
        String fileName = getFileNameForItem(item); 
        ItemWriter<? super T> writer = delegates.get(fileName);
        if(writer == null) {
            // There is no writer for the fileName. 
            //create one
            writer = createMyItemWriter(fileName);
            delegates.put(fileName, writer);
        }
        return writer;
    }
    
    ItemWriter<? super T> createMyItemWriter(String fileName) {
        // create the writer. Maybe a org.springframework.batch.item.xml.StaxEventItemWriter 
        // set the resource(fielName)
        //open the writer
    }
    
    
    
    
    // Identify the name of the target file - toto_foo.xml, tata_foo.xml etc
    private String getFileNameForItem(Item item) {
        .....
    }
    
    @Override
    public void close() throws ItemStreamException {
        for (ItemWriter<? super T> writer : delegates) {
            if (!ignoreItemStream && (writer instanceof ItemStream)) {
                ((ItemStream) writer).close();
            }
        }
    }
    
    @Override
    public void open(ExecutionContext executionContext) throws ItemStreamException {
        // Do nothing as we do not have any writers to begin with. 
        // Writers will be created as needed. And will be opened after creation. 
    }
    
    @Override
    public void update(ExecutionContext executionContext) throws ItemStreamException {
        for (ItemWriter<? super T> writer : delegates) {
            if (!ignoreItemStream && (writer instanceof ItemStream)) {
                ((ItemStream) writer).update(executionContext);
            }
        }
    }