javascriptmediastream

Procedural Audio using MediaStreamTrack


I want to encode a video (from a canvas) and add procedural audio to it.

The encoding can be accomplished with MediaRecorder that takes a MediaStream.

For the stream, I want to obtain the video part from a canvas, using the canvas.captureStream() call.

I want to add an audio track to the stream. But instead of microphone input, I want to generate the samples for those on the fly, for simplicity sake, let's assume it writes out a sine-wave.

How can I create a MediaStreamTrack that generates procedural audio?


Solution

  • The Web Audio API has a createMediaStreamDestination() method, which will return a MediaStreamAudioDestinationNode object, on which you'll be able to connect your audio context, and which will give you access to a MediaStream instance fed by the audio context audio output.

    document.querySelector("button").onclick = (evt) => {
      const duration = 5;
      evt.target.remove();
      const audioContext = new AudioContext();
      const osc = audioContext.createOscillator();
    
      
      const destNode = audioContext.createMediaStreamDestination();
      const { stream } = destNode;
      
      osc.connect(destNode);
      osc.connect(audioContext.destination);
      osc.start(0);  
      osc.frequency.value = 80;
      osc.frequency.exponentialRampToValueAtTime(440, audioContext.currentTime+10);
      osc.stop(duration);
      
      // stream.addTrack(canvasStream.getVideoTracks()[0]);
      const recorder = new MediaRecorder(stream);
      const chunks = [];
      recorder.ondataavailable = ({data}) => chunks.push(data);
      recorder.onstop = (evt) => {
        const el = new Audio();
        const [{ type }] = chunks; // for Safari
        el.src = URL.createObjectURL(new Blob(chunks, { type }));
        el.controls = true;
        document.body.append(el);
      };
      recorder.start();
      setTimeout(() => recorder.stop(), duration * 1000);
      console.log(`Started recording, please wait ${duration}s`);
     };
    <button>begin</button>