I'm trying to build a file watcher that copies files when they are finally added using chokidar. The files have differing sizes, but range between 200MB and 1GB.
watch(`${inputFolder}/*/**`, {
ignoreInitial: false,
awaitWriteFinish: true,
}).on("add", (path) => {
handleFileAdded(path);
retryFailedPaths();
});
I'm having a weird issue where some files are only being copied partly and then left there without any errors. The target size differs and is not constant. I get the feeling that somehow the process is being interrupted by something. Strangely the code after the (corrupted) copying happily continues, in this case remove the source file.
fs.copyFileSync(sourcePath, newPath, fs.constants.COPYFILE_FICLONE);
fs.unlinkSync(sourcePath);
From that I conclude that the process itself is not being killed. I'm also observing that chokidar fires faster than previous files could copy but I assumed that not being a problem since the files have nothing to do with each other.
I tried fs.copySync and fs.promises.copy, both of which have the same behaviour. I couldn't reproduce this locally, most likely because my computer is faster than the target environment. I also had a look at resource usage on the target device but it never reached the cap neither RAM- nor CPU-wise. On the target device the files are being copied from one hard drive to another though.
Here are some extra information:
environment: docker
runtime: node14:alpine
The solution is pretty easy. The file was added and registered by chokidar before it was completely written to disk. That way, the copyFileSync overwrote the target file anytime a new "update" was registered by chokidar.