spring-integrationpoller

Spring Integration:prevent duplicates in multi node environment


I have a file which is polled by inbound adapter. Environment wise, I have multiple nodes polling a single file which is on a NAS location(visible to all) and prevent duplicates is true.

Surprisingly, if i put a file repeatedly, none of the nodes pick the file until server is restarted.

Assuming that spring integration used to keep this information in spring context, this should have been picked by other nodes if missed by one.

Can somebody explain where this flag or meta data is maintained?


Solution

  • See the documentation. You need a FileSystemPersistentAcceptOnceFileListFilter with a shared MetadataStore.

    By default, a memory-based AcceptOnceFileListFilter is used, which does not take into account the file modified time.

    The AcceptOnceFileListFilter stores its state in memory. If you wish the state to survive a system restart, consider using the FileSystemPersistentAcceptOnceFileListFilter instead. This filter stores the accepted file names in a MetadataStore implementation (Section 9.5, “Metadata Store”). This filter matches on the filename and modified time.