javascriptcanvasvideohtml5-canvas

Optimizing transparent video method

enter image description here Video

Hello, I've been trying to render this video on my site with a transparent background. After some research I managed to stumble upon this resource which let me get my desired effect using a canvas.

Using the B&W mask on the right side of the video, I can set the pixels I'd like to be transparent. Issue is my current method at times stutters or has low framerate. How would I go about optimizing this or taking a different approach entirely? I feel like having a second canvas just to draw then read the pixels seems inefficient.

Also to note, for whatever reason my rendered image seems to be squished compared to the original.

enter image description here

Here is my current code

const canvas = this.$refs.canvas.getContext("2d", { willReadFrequently: true })
const render = this.$refs.render.getContext("2d", { willReadFrequently: true })
const video = this.$refs.video

video.play()

function maskVideo() {
    canvas.drawImage(video, 0, 0, 3840, 1280)
    
    const frame = canvas.getImageData(0, 0, 1920, 1280)
    const frameData = frame.data

    for (var i = 0; i < frameData.length; i += 4) {
        const [r, g, b] = [frameData[i], frameData[i + 1], frameData[i + 2]]

        frameData[i + 3] = (r + g + b)
    }

    render.putImageData(frame, 0, 0)

    window.requestAnimationFrame(maskVideo)
}

window.requestAnimationFrame(maskVideo)
Solution

  • But if you insist that it must be done manually (with no file re-encoding) then...
    For smoother playback of such masked videos (at 3840 x 1280) you need to use the GPU.

    Compare your Canvas speed with my example GPU demo:
    https://vcone.github.io/public/demos/js/draw_vid_mask/index.html

    Is GPU processing (3D) looking smoother than the drawImage() of Canvas (2D)?
    You can also check the page's source code to learn from it.
    It is based on this other StackOverflow Answer.

    In that Answer's code you need to modify the function main(void) to achieve a masking effect.
    First create two variables to store the current colours of video and mask pixels separately (because easier to process later).

    mediump vec4 my_FragColor_video = vec4(1.0, 1.0, 1.0, 1.0);
mediump vec4 my_FragColor_mask = vec4(1.0, 1.0, 1.0, 1.0);

void main(void) 
{
    if( (vDirection.x * 0.5) < 0.005) //# when GPU is scanning at first half of image size 
    {
        //# read from first half
        my_FragColor_video = texture2D(uSampler, vec2( (vDirection.x * 0.5 ) + 0.5 , (vDirection.y * 0.5 ) + 0.5) );
        
        //# read from second half
        my_FragColor_mask = texture2D(uSampler, vec2( (vDirection.x * 0.5 ) + 1.0 , (vDirection.y * 0.5 ) + 0.5) );
    }
    
    //# first put te video's pixel colour
    gl_FragColor = my_FragColor_video;
    
    //# then alpha is from using "green" of mask, but can use any r/g/b channel of mask 
    gl_FragColor.a = my_FragColor_mask.g; 
    
}

    Hope it helps as a starting point for smoother video playback on your project.

    Experiment by editing the GPU main() code to work with your screen size.
    Expeminent which vars will adjust picture w/h, scaling, reading position, etc.

    If you're new to GPU coding then just remember: