apache-camelcamel-ftpcamel-fileapache-camel-file

Camel SFTP how to prevent processing the same file incase of multiple nodes?

This is my route

    from("sftp://userName:password@ip:22/<my-folder>?move=.done")
        .routeId("my-route-1")
        .<processing-logic>

How to avoid processing the same files incase of multiple instances?

Solution

  • Camel recently introduced some interesting clustering capabilities - see here.

    In your particular case, you could model a route which is taking the leadership when starting the directory polling, preventing thereby other nodes from picking the (same or other) files.

    Set it up is very easy, and all you need is to prefix singleton endpoints according to the master component syntax: master:namespace:delegateUri

    This would result in something like this:

    from("master:mycluster:sftp://...")
        .routeId("clustered-route")
        .log("Clustered sftp polling !");