javascripthtmlhtml5-canvashtml5-videopicture-in-picture

Get mouse events from floating picture-in-picture window

I have a canvas that is being rendered onto a video via video.srcObject = canvas.captureSteam()

That video is then made the active picture-in-picture element using video.requestPictureInPicture()

I would like to add some interactivity to the floating window (for example, a clickable button rendered in the canvas).

Is there a way to get mouse events on the floating window? All I really need is the mouse position and mouse button down/up.

I have tried using addEventListener on the PictureInPictureWindow that is returned from requestPictureInPicture but it seems like the only event that it is able to pass is resize

Solution

  • So for my use case, this was for a video conferencing app. I only needed audio and video mute and a hangup button; it turns out that these are implemented in the MediaSession API.

    Adding the following made audio mute/unmute, video mute/unmute, and a hangup button show on the picture-in-picture window on hover:

    navigator.mediaSession.setActionHandler('togglemicrophone', toggleAudioMuted);
navigator.mediaSession.setActionHandler('togglecamera', toggleVideoMuted);
navigator.mediaSession.setActionHandler('hangup', hangup);

    navigator.mediaSession.setMicrophoneActive and navigator.mediaSession.setCameraActive could also be used to sync the muted/unmuted status with the picture-in-picture window.

    More details can be found here: https://w3c.github.io/mediasession/#the-mediasession-interface

    And here: https://developer.mozilla.org/en-US/docs/Web/API/MediaSession