node.jsffmpegfluent-ffmpeg

Running ffmpeg in a while loop causes memory leak


I'm trying to run the following ffmpeg commands in a while loop synchronously.

Each time round the loop I wait for the current command to finish before I move on the next one. However I'm getting unexpected results and then it crashes.

  import ffmpegPath from '@ffmpeg-installer/ffmpeg'
  import ffmpeg from 'fluent-ffmpeg'

  ffmpeg.setFfmpegPath(ffmpegPath.path)

  const command = ffmpeg()

  let VIDEOS = 1000
  let videoIdx = 1

  while (videoIdx <= VIDEOS) {
    await new Promise((resolve) => {
      command
        .on('end', () => {
          setTimeout(() => {
            console.log(`${videoIdx}/${VIDEOS}`)
            videoIdx++
            resolve()
          }, 100)
        })
        .on('error', () => {
          console.log('error = ', error)
        })
        .input(`MyExernalHardDrive/input/video-${videoIdx}-frame-%d.png`)
        .inputFPS(1/0.0425)
        .output(`MyExernalHardDrive/output/video-${videoIdx}.mp4`)
        .outputFPS(24)
        .noAudio()
        .run()
    })
  }

In the console I'm expecting to see:

1/1000

then

2/1000

then

3/1000

etc..

Instead I'm getting logs in batches like:

1/1000
2/1000
3/1000

then

4/1000
5/1000
6/1000
7/1000

then it keeps incrementing but I get them in even bigger batches:

45/1000
46/1000
47/1000
48/1000
49/1000
50/1000
51/1000
52/1000
53/1000
54/1000
55/1000

And then I get:

(node:14509) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 
11 end listeners added to [FfmpegCommand]. 
Use emitter.setMaxListeners() to increase limit

I then exit the process and these are the files that have been created:

video-1.mp4
video-2.mp4
video-4.mp4
video-7.mp4
video-11.mp4
video-16.mp4
video-22.mp4
video-29.mp4
video-37.mp4
video-46.mp4
video-56.mp4
video-67.mp4

So it seems that:

  1. The timing is all over the place and not sequential at all
  2. Processes are piling up until there is a memory leak

Why is this happening since I'm using await before I move on to the next process and how do I fix this ?


Solution

  • // ...
      const command = ffmpeg()
    
      let VIDEOS = 1000
      let videoIdx = 1
    
      while (videoIdx <= VIDEOS) {
        await new Promise((resolve) => {
          command
            .on('end', () => {
              setTimeout(() => {
                console.log(`${videoIdx}/${VIDEOS}`)
                videoIdx++
                resolve()
              }, 100)
            })
    // ...
    

    The ffmpeg instance in variable command is being reused in every loop. It's an Event Emitter and shouldn't be reused, because some events only fire once, like "end". The "end" event on this single instance receives many listeners. The Node runtime is giving a warning, because this is usually not something you want (if it is, you can opt-in to disable the limit).

    The solution is to not re-use the same instance:

    // ...
      let VIDEOS = 1000
      let videoIdx = 1
    
      while (videoIdx <= VIDEOS) {
        await new Promise((resolve) => {
          ffmpeg()
            .on('end', () => {
              setTimeout(() => {
                console.log(`${videoIdx}/${VIDEOS}`)
                videoIdx++
                resolve()
              }, 100)
            })
    // ...
    

    Now every loop iteration a fresh instance with a clean slate will be created.