I'm wanting to watch an upload folder for new files, and trigger a script when anything is uploaded. for this purpose I have installed watchman on my CentOS 7 box, and set it up to watch the upload folder. This works, but when a large file is being uploaded watchman will trigger one or more times before the upload is complete. Since my script will be moving the file, this can result in corrupted data and failed uploads. How can I filter out these "partial" triggers?
As an example, a test script I made that just dumped information to a file gave me this output during a single upload:
Wed Sep 18 08:39:20 AKDT 2019 - test.nc STDIN: [{"mode": 33188, "oclock": "c:1568822800:18913:1:743734", "exists": true, "new": true, "size": 5924978688, "name": "test.nc"}]
Wed Sep 18 08:39:22 AKDT 2019 - test.nc STDIN: [{"mode": 33188, "oclock": "c:1568822800:18913:1:747283", "exists": true, "new": false, "size": 6056411136, "name": "test.nc"}]
Wed Sep 18 08:39:22 AKDT 2019 - test.nc STDIN: [{"mode": 33188, "oclock": "c:1568822800:18913:1:747324", "exists": true, "new": false, "size": 6057754624, "name": "test.nc"}]
Wed Sep 18 08:39:24 AKDT 2019 - test.nc STDIN: [{"mode": 33188, "oclock": "c:1568822800:18913:1:752502", "exists": true, "new": false, "size": 6229433544, "name": "test.nc"}]
I was able to work around this issue by adjusting the "settle" parameter. Apparently the default of 20 ms is too low for network transfers, resulting in periods where the disk is idle as incoming data is buffered or something. By bumping this setting up to 500, watchman no longer triggers before the file transfer is complete.