apache-camelsambacamel-ftp

Using camel-smb SMB picks up (large) files while still be written to

When trying to create cyclic moving of files encountered strange behavior with readLock. Create a large file (some 100Mb's) and transfer it using SMB from out to in folder.

FROM:

smb2://smbuser:****@localhost:4455/user/out?antInclude=FILENAME*&consumer.bridgeErrorHandler=true&delay=10000&inProgressRepository=%23inProgressRepository&readLock=changed&readLockMinLength=1&readLockCheckInterval=1000&readLockTimeout=5000&streamDownload=true&username=smbuser&delete=true

TO:

smb2://smbuser:****@localhost:4455/user/in?username=smbuser

Create another flow to move the file back from IN to OUT folder. After some transfers the file will be picked up while still being written to by another route and a transfer will be done with a much smaller file, resulting in a partial file at the destination.

FROM:

smb2://smbuser:****@localhost:4455/user/in?antInclude=FILENAME*&delete=true&readLock=changed&readLockMinLength=1&readLockCheckInterval=1000&readLockTimeout=5000&streamDownload=false&delay=10000

TO:

smb2://smbuser:****@localhost:4455/user/out

Question is: why my readLock is not working properly (p.s. streamDownload is required)?

UPDATE: turns out this only happens on windows samba share, and with streamDownload=true. So, something with stream chunking. Any advice welcome.

Solution

  • The solution requires to prevent polling strategy from automatically picking up a file, and become aware of readLock (in progress) on the other side. So I lowered delay to 5 seconds and in FROM part, on both sides, I added readLockMinAge to 5s which will inspect file modification time. Since streaming goes for every second this is enough time to prevent read lock.

    An explanation of why the previously mentioned situation happens:

    1. When a route prepares to pick-up from out folder, a large file (1GB) is in progress chunk by chunk to in folder. At the end of the streaming file is marked for removal by camel-smbj and file receive status STATUS_DELETE_PENDING.
    2. Now another part of this process starts to send a newly arrived file to the out folder and finds out that this file already exists. Because of the default fileExists=Override strategy It tries to delete (afterward store) an existing file (which is still not deleted from the previous step) and receives an exception which causes some InputStream chunks to be lost.