node.jssetintervalchokidar

Collect changes in a file every fixed time in node js


I have an external program that is streaming data to a csv file every now and then (but quit a lot). I want to collect every 10 seconds all the changed data and do some processing on it. means I want to process only lines I didn't processed before.

this is the basic code:

function myFunction() {

  var loop = setInterval(
    () =>
      {
          var instream = fs.createReadStream("rawData.csv"); //should somehow include only new data since last cycle
          var outstream = fs.createWriteStream("afterProcessing.csv");
          someProcessing(instream, outstream);
          outstream.on('finish', () => {
               sendBackResults("afterProcessing.csv");
                });
        //will exit the loop when 'run' flag will change to false
          if(!run) ? clearInterval(loop) : console.log(`\nStill Running...\n`) ;
      } , 10000 );    

 }

Now, I tried to work with chokidar and fs.watch but I couldn't figure out how to use them in this case.


Solution

  • fs.createReadStream can take a start parameter

    options can include start and end values to read a range of bytes from the file instead of the entire file. Both start and end are inclusive and start counting at 0

    So you need to save the last position read, and use it on start.

    You can get that using: instream.bytesRead.

    let bytesRead = 0;
    
    instream.on('end', () => {
      bytesRead = instream.bytesRead;
    });