javascriptreactjsffmpegwebrtchttp-live-streaming

How to make media recorder api individual chunks playable by it self


I'm trying to send individual chunks to the server instead of sending the whole chunks at once. This way, I have Ffmpeg on my Rails server to convert these chunks to HLS and upload them to S3 to stream the video instantly. However, I've encountered an issue the Media Recorder only provides playable chunks for the first segment after that, they are not playable and need to be concatenated to play.

To avoid this, I've taken a different approach where I start a new Media Recorder every 3 seconds so that I get a playable chunk every time. However, this approach has its issues the video glitches a bit due to the delay when I stop and start a new Media Recorder. Is there a way to achieve this with ease? This is my current status. please help!

const startVideoRecording = async (
    screenStream: MediaStream,
    audioStream: MediaStream
  ) => {
    setStartingRecording(true);

    try {
      const res = await getVideoId();
      videoId.current = res;
    } catch (error) {
      console.log(error);
      return;
    }

    const outputStream = new MediaStream();
    outputStream.addTrack(screenStream.getVideoTracks()[0]);
    outputStream.addTrack(audioStream.getAudioTracks()[0]); // Add audio track

    const mimeTypes = [
      "video/webm;codecs=h264",
      "video/webm;codecs=vp9",
      "video/webm;codecs=vp8",
      "video/webm",
      "video/mp4",
    ];

    let selectedMimeType = "";
    for (const mimeType of mimeTypes) {
      if (MediaRecorder.isTypeSupported(mimeType)) {
        selectedMimeType = mimeType;
        break;
      }
    }

    if (!selectedMimeType) {
      console.error("No supported mime type found");
      return;
    }

    const videoRecorderOptions = {
      mimeType: selectedMimeType,
    };

    let chunkIndex = 0;

    const startNewRecording = () => {
      // Stop the current recorder if it's running
      if (
        videoRecorderRef.current &&
        videoRecorderRef.current.state === "recording"
      ) {
        videoRecorderRef.current.stop();
      }

      // Create a new MediaRecorder instance
      const newVideoRecorder = new MediaRecorder(
        outputStream,
        videoRecorderOptions
      );
      videoRecorderRef.current = newVideoRecorder;

      newVideoRecorder.ondataavailable = async (event) => {
        if (event.data.size > 0) {
          chunkIndex++;
          totalSegments.current++;
          const segmentIndex = totalSegments.current;
          console.log(event.data);
          handleUpload({ segmentIndex, chunk: event.data });
        }
      };

      // Start recording with a 3-second interval
      newVideoRecorder.start();
    };

    // Start the first recording
    startNewRecording();

    // Set up an interval to restart the recording every 3 seconds
    recordingIntervalIdRef.current = setInterval(() => {
      startNewRecording();
    }, 3000);

    setIsRecording(true);
    setStartingRecording(false);
  };

Solution

  • That is largely how it is designed to work, for efficient video compression you need key frames followed by a large number of delta frames that only describe the difference.

    MediaRecorder allows tuning the interval (or frame count) between those key frames as part of the media recorder options. You will still need to concatenate blobs to ensure they are starting with a keyframe.