node.jsstreamthrough2

Converting WriteStream to TransformStream


I have a (somewhat weird) writable stream that I need to convert to a transform stream.

The writable stream, normally, sits at the end of a pipe chain and emits custom events once it has collected enough data for its output. I want it to go in the middle so I can pipe it to another writeStream, i.e:

    readStream.pipe(writeStreamToConvert).pipe(finalWriteStream);

What I done is the following and it works.

    const through2 = require('through2')


    var writeStreamToConvert = new WriteStreamToConvert();

    return through2.obj(function (chunk, enc, callback) {

        writeStreamToConvert.write(chunk)

        // object is the event emitted from the writestream
        writeStreamToConvert.on('object', (name, obj ) => {
            this.push(JSON.stringify(obj, null, 4) + '\n')
        });            

        callback()
    })

This works fine, does not seem to leak memory and is fairly quick. However node gives me a warning:

Warning: Possible EventEmitter memory leak detected. 11 object listeners added. Use emitter.setMaxListeners() to increase limit

So I am a little bit curious if this is the correct way of converting writestreams?


Solution

  • The event handler would be best placed in a Transform stream constructor. Since through2 does not support such initialization, you would need to use node's stream API directly.

    Currently, a new event handler (which is never removed -- that is how .on() works) is being added for every object written to the through2 stream. That is why the warning occurs.