node.jslarge-files

Writing large files with Node.js


I'm writing a large file with node.js using a writable stream:

var fs     = require('fs');
var stream = fs.createWriteStream('someFile.txt', { flags : 'w' });

var lines;
while (lines = getLines()) {
    for (var i = 0; i < lines.length; i++) {
        stream.write( lines[i] );
    }
}

I'm wondering if this scheme is safe without using drain event? If it is not (which I think is the case), what is the pattern for writing an arbitrary large data to a file?


Solution

  • That's how I finally did it. The idea behind is to create readable stream implementing ReadStream interface and then use pipe() method to pipe data to writable stream.

    var fs = require('fs');
    var writeStream = fs.createWriteStream('someFile.txt', { flags : 'w' });
    var readStream = new MyReadStream();
    
    readStream.pipe(writeStream);
    writeStream.on('close', function () {
        console.log('All done!');
    });
    

    The example of MyReadStream class can be taken from mongoose QueryStream.