I have an external program that is streaming data to a csv file every now and then (but quit a lot). I want to collect every 10 seconds all the changed data and do some processing on it. means I want to process only lines I didn't processed before.
this is the basic code:
function myFunction() {
var loop = setInterval(
() =>
{
var instream = fs.createReadStream("rawData.csv"); //should somehow include only new data since last cycle
var outstream = fs.createWriteStream("afterProcessing.csv");
someProcessing(instream, outstream);
outstream.on('finish', () => {
sendBackResults("afterProcessing.csv");
});
//will exit the loop when 'run' flag will change to false
if(!run) ? clearInterval(loop) : console.log(`\nStill Running...\n`) ;
} , 10000 );
}
Now, I tried to work with chokidar and fs.watch but I couldn't figure out how to use them in this case.
fs.createReadStream
can take a start
parameter
options can include start and end values to read a range of bytes from the file instead of the entire file. Both start and end are inclusive and start counting at 0
So you need to save the last position read, and use it on start
.
You can get that using: instream.bytesRead
.
let bytesRead = 0;
instream.on('end', () => {
bytesRead = instream.bytesRead;
});