I would like to create a video file from multiple images uploaded to my site.
Until now, what I do is take these images, draw them 1-by-1 on a canvas, and use the MediaRecorder API to record them. However, there is a lot of idle time.
Instead, I want to use the VideoEncoder API.
I created an encoder that saves every chunk as a buffer:
const chunks = [];
let encoder = new VideoEncoder({
output: (chunk) => {
const buffer = new ArrayBuffer(chunk.byteLength)
chunk.copyTo(buffer);
chunks.push(buffer);
},
error: (e) => console.error(e.message)
});
And configured it with my settings:
encoder.configure({
codec: 'vp8',
width: 256,
height: 256,
bitrate: 2_000_000,
framerate: 25
});
Then, I encode every image as a frame:
const frame = new VideoFrame(await createImageBitmap(image));
encoder.encode(frame, {keyFrame: true});
frame.close();
And finally, I try to create a video from it:
await encoder.flush();
const blob = new Blob(chunks, {type: 'video/webm; codecs=vp8'});
const url = URL.createObjectURL(blob);
However, that URL blob is unplayable. If I try to download it, VLC does not show it. If I set it as the source for a video element, I get:
DOMException: The element has no supported sources.
How do I encode multiple frames into a video that is playable?
How do I know which codecs / blob types are supported?
The following codepen is the above code, concatenated and joined into a single function. https://codepen.io/AmitMY/pen/OJxgPoG?editors=0010
VideoEncoder and other classes from the WebCodecs API provide you with the way of encoding your images as frames in a video stream, however encoding is just the first step in creating a playable multimedia file. A file like this may potentially contain multiple streams - for instance when you have a video with sound, that's already at least one video and one audio stream, so a total of two. You need additional container format to store the streams so that you do not have to send the streams in separate files. To create a container file from any number of streams (even just one) you need a multiplexer (muxer for short). Good summary of the topic can be found in this Stack Overflow answer, but to quote the important part:
- When you create a multimedia file, you use a coder algorithms to encode the video and audio data, then you use a muxer to put the streams together into a file (container). To play the file, a demuxer takes apart the streams and feeds them into decoders to obtain the video and audio data.
- Codec means coder/decoder, and is a separate concept from the container format. Many container formats can hold lots of different types of format (AVI and QuickTime/MOV are very general). Other formats are restricted to one or two media types.
You may think "i have only one stream, do i really need a container?" but multimedia players expect received data (either data read from a file or streamed over network) to be in a container format. Even if you have only one video stream, you still need to pack it into a container for them to recognize it.
Joining the byte buffers into one big blob of data will not work:
const blob = new Blob(chunks, {type: 'video/webm; codecs=vp8'});
Here you try to glue all the chunks together and tell the browser to interpret it as a WebM video (video/webm MIME type) but it can't do it, as it is not properly formatted. This in turn is the source of the error. To make it work, you have to append relevant metadata to your chunks (usually formated as buffers of binary data with specific format depending on the type of a container as well as codec) and pass it to a muxer. If you use a library for muxing that is designed to work with raw streams of video (for example, those coming from WebCodecs API) then it will probably handle the metadata for you. As a programmer you most likely will not have to deal with this manually, however if you want to understand more about the whole process then i suggest you read about metadata present in various container formats (for example, VC.Ones comments below this answer).
Sadly, muxers do not seem to be a part of the WebCodecs API as of now. Example in the official repository of the API uses the muxAndSend() function as the encoder output callback:
const videoEncoder = new VideoEncoder({
output: muxAndSend,
error: onEncoderError,
});
And above in the code we can see that this function needs to be supplied by the programmer (original comments):
// The app provides a way to serialize/containerize encoded media and upload it.
// The browser provides the app byte arrays defined by a codec such as vp8 or opus
// (not in a media container such as mp4 or webm).
function muxAndSend(encodedChunk) { ... };
Here is a link to a discussion about adding muxing support to browsers and here is an issue in the official repo tracking this feature. As of now, there does not seem to be a built in solution for your problem.
To solve it you could possibly use a third party library such as mux.js or similar (here is a link to their "Basic Usage" example which may help you). Alternatively, this project claims to create WebM containers out of VideoEncoder encoded data. This excerpt from the description of their demo seems to be exactly what you wanted to achieve (except with a webcam as the VideoFrame source, instead of a canvas):
When you click the Start button, you’ll be asked by the browser to give permission to capture your camera and microphone. The data from each is then passed to two separate workers which encode the video into VP9 and audio into Opus using the WebCodecs browser API.
The encoded video and audio from each worker is passed into a third worker which muxes it into WebM format.
I cannot provide you with a code sample as i have not used any of mentioned libraries myself, but i am sure that after understanding the relation between encoders and muxers you should be able to solve the problem on your own.
EDIT: I have found another library which might help you. According to their README:
What's supported:
- MP4 video muxing (taking already-encoded H264 frames and wrapping them in a MP4 container)
- MP4/H264 encoding and muxing via WebCodecs
Many libraries and sources i find online seem to be WASM-based, usually implemented in C or another language compiling to native machine code. This is probably due to the fact that large libraries exist (first thing that comes to mind is ffmpeg) which deal with all sorts of media formats, and this is what they are written in. JS libraries are often written as bindings to said native code to avoid reinventing the wheel. Additionally, i would assume that performance may also be a factor.
Disclaimer: While you used video/webm as the MIME type in your code sample, you did not explicitly state what file format do you want your output to be, so i allowed myself to reference some libraries which produce other formats.
EDIT 2:
David Kanal's answer below provides another example of a library which could be used for muxing WebM.