video-capturegetusermediavideo-recordingweb-mediarecorder

How to stop pixelated artifacts from mediarecorder?

We are using the browser's MediaRecorder to record the user's camera. I can't figure out why some cameras record artifacts somewhat frequently. It doesn't happen with all cameras.

Here are some screenshots of what happens...

Here is the normal image from the video enter image description here

Then for a split second it gets pixelated and then goes back to looking like previous image. enter image description here

Here is an excerpt of a video that has this artifact https://drive.google.com/file/d/1SYMBRjMlvOTO-LnlG5HLX0ncN3NyUXcc/view?usp=sharing

recorder = new MediaRecorder(local_media_stream, {
           mimeType: encoding_options,
           audioBitsPerSecond: 96000,
           videoBitsPerSecond: 8000000,
         });
         recorder.ondataavailable = function(e) {
             that.save_blob(e.data, blob_index);
             blob_index++;
         }
         recorder.start(15000)

For the local_media_stream Im just grabbing the local audio and video

    navigator.mediaDevices.getUserMedia({ audio: {
        deviceId: { exact: currentMic.deviceId},
        channelCount: 1,
        noiseSuppression: options.echo_cancellation,
        echoCancellation: options.echo_cancellation,
      },
       video: {
         width: { ideal: $("#subscription_id").data("max-width") },
         height: { ideal: $("#subscription_id").data("max-height") },
         frameRate: 30,
         deviceId: { exact: currentCam.deviceId }
       }
     })
Solution

  • It's usually not the camera that creates these video compression artifacts. They're in the nature of video compression, and somewhat dependent on the computer's power. Most frames of video are so-called "interframes", holding the compressed difference between one frame and the next. To decode an interframe requires decoding all the frames that came before it.

    Those sequences of interframes start with "intraframes", also known as Instantaneous Decoder Refresh (IDR) frames. Intraframes can be decoded standing alone. But there's a tradeoff: they require more compressed video data.

    When you ask for 30 frames/second you give an implicit limit on how much data can fit in each frame. And sometimes the intraframe needs to have its quality reduced so its data fits in the time interval.

    My point: the blockiness you see are the intraframes.

    What can you do about this? Honestly, MediaRecorder doesn't give much control over the way the video encoder works. So, the things you can tweak are fairly minor. They are:

    Keep in mind that people who compress video for online streaming use big fat servers, and that it can ten minutes or more on those servers to compress one minute of video. They often use ffmpeg, which has hundreds of ways of tweaking the compression. MediaRecorder does it in real time.

    Also, webcam hardware is quite variable in quality. Bad webcams have Coke-bottle shards for lenses and crummy sensors. Good ones are very good. None are up to the standard of professional video equipment. But that equipment costs thousands.

    And, keep in mind that compressed video is more JPEG-like than PNG-like. It's not perfect for encoding high-contrast static pictures.

    High quality digital video recording is a well-developed subspeciality of computing. If you want "the best audio and video possible" you won't be using a web browser / Javascript / getUserMedia / MediaRecorder. You'll be using a native (downloaded and installed) app like OBS Studio. It gives you a lot more control over the capture and compression process.

    If you use capture / compression technology embedded in browsers in early 2021 you'll be, practically, limited to a couple of megabits a second and some resolution constraints.