I need to migrate some data from one table to another table applying some processing in between. Processing is proprietary and is exposed as REST service. So I need to read records from a table call REST services and then write processed record in another table. How do I implement this as Spring Batch so that I can make REST calls in parallel for multiple records because that's where I am expecting a lot of time for each record? ItemProcessor where I am planning to make the REST call only accept a single Item in process method.
When the processing is the bottleneck, you can use:
AsyncItemProcessor:this will process items asynchronously and can help in scaling out the processing. See more details about this component here: https://docs.spring.io/spring-batch/4.0.x/reference/html/spring-batch-integration.html#asynchronous-processors
Remote chunking: with this scalability option, you send data to remote workers for processing. This helps you scale the processing horizontally by adding more workers as needed. More details about remote chunking in this section: https://docs.spring.io/spring-batch/4.0.x/reference/html/spring-batch-integration.html#remote-chunking
There are some important considerations you would want to pay attention to when choosing a scalability option, they are detailed here: https://stackoverflow.com/a/20342308/5019386