apache-cameljmsintegrationcamel-ftpheader-enricher

How to enrich message in File Route from SFTP to JMS in clustered environment using Apache Camel?


We are going to read a file from SFTP and put employees in database.

Following is XML Structure

 <employees>
     <employee></employee>
     <employee></employee>
     <employee></employee>
 <employees>    

The strategy i think is:

  1. Pick XML file from SFTP ->
  2. Fetch Employee No. from XML -> (24000 Employees)
  3. Fetch Data from "System 1 and System 2" on base of employee no. in xml file all together in memory ->
  4. Split Employee from XML ->
  5. Assign Data from "System 1 and System 2" to Each employee Exchange ->
  6. Put Each employee XML to JMS Queue (Read in clustered envionment)

Is there any other best strategy to handle this scenario in Apache Camel in abve route or route 2 (Jms Queue)?

Note: As it is clustered environment; we can't fetch data from System1 & System2 in servers itself then we need to keep tracking by assigning batch id to one file contents. You can think this scenario in other integrations too


Solution

  • The above steps can be improved by

    1. Speed up your process after step 4 by parallel processing (e.g. use threadpool)
    2. Fasten SFTP process by release core thread after step 3 (e.g. use wireTap)
      • To prevent data lost, store exchange to another persistent storage (e.g. JMS queue)
      • The application's memory usage will increase

    If step 3 (fetch data from "System 1 and System 2") can be done concurrently, launch multiple application instance with idempotent (leverage external cache or DB) might help for lots of files.