canvasvideo-streamingwebrtcmediastreamkurento-media-server

Merging the canvas stream with getUserMedia's audio stream is not audible (kurento/webrtc)


When the user does not have a video device, I'm using the canvas video stream and merging the video track with the stream obtained from getUserMedia(audio stream). Peer connects perfectly but other users are unable to hear the audio.

If I share my screen and merge this video track into the stream, then the audio works perfectly. I noticed that CanvasMediaStream is not working but MediaStream is working perfectly. I don't know if this is an issue at kurento's end or if I am doing something wrong.


Solution

  • No Kurento to test, but the problem is probably that the CanvasCaptureMediaStreamTrack gets muted after some time of inactivity on the canvas's context.

    To workaround that you can simply set up a drawing loop that will update the canvas regularly (every half seconds should be largely enough without causing too much overhead either).

    Also, you may want to start from a fresh MediaStream, though I doubt this has any influence:

    // assumes there is a 'canvas' and a 'mic_stream'
    
    // make the context active, so the stream is not muted
    const ctx = canvas.getContext("2d");
    setInterval(() => ctx.clearRect(0,0,1,1), 500);
    ctx.clearRect(0,0,1,1);
    const canvas_stream = canvas.captureStream();
    
    const canvas_track = canvas_stream.getVideoTracks()[0];
    const mic_track = mic_stream.getAudioTracks()[0];
    const merged_stream = new MediaStream([ canvas_track, mic_track ]);
    
    // do something with 'merged_stream'