I have code that records the stream of a video manipulated and displayed on a canvas element and combines it with the AudioContext stream of the same video to give it sound.
It works 100% of the time on Desktop. However, on mobile (Safari & Chrome) it works about 50% of the time when it has the AudioContext attached and 100% of the time when it does not.
I am using Angular 15 / Ionic 6.
Here is my code. I set up the MediaStream by adding the AudioContext stream and the video canvas stream as tracks. Then I use this stream in a MediaRecorder. When I start the MediaRecorder I begin to manipulate each frame of the video and display it on the canvas until the MediaRecorder seems to automatically stop when the video ends.
HTML (the click event triggers processing and connection of the AudioContext):
<ion-button (click)="startVideoProcessing()">Process Video</ion-button>
<video id="uploadedVideo" #uploadedVideo autoplay webkit-playsinline playsinline></video>
<canvas id="canvasVideo" #canvasVideo></canvas>
<canvas id="segmentationMaskCanvasVideo" #segmentationMaskCanvasVideo></canvas>
Setting up the MediaRecorder:
async videoProcessing() {
// audioContext: AudioContext;
// sourceNode: MediaElementAudioSourceNode;
this.stream = new MediaStream(
[...this.videoCanvas.captureStream(30).getVideoTracks(),
this.getAudioTrack()
]);
const options = {
audioBitsPerSecond: 128000,
videoBitsPerSecond: 2500000,
mimeType: this.mimeType,
};
// recorder: MediaRecorder
this.recorder = new MediaRecorder(this.stream, options);
this.recorder.start();
this.recorder.ondataavailable = (event) => {
if (event.data && event.data.size > 0) {
this.chunks.push(event.data);
}
};
this.recorder.onerror = (event) => {
console.error(event);
throw new Error(event.toString());
};
this.recorder.onstop = () => {
const videoBlob = new Blob(this.chunks, { type: this.mimeType });
const videoFile = new File([videoBlob], this.mimeType === 'video/mp4' ? 'video-green-screen.mp4' : 'video-green-screen.webm', { type: this.mimeType });
this.fileSelected.emit(videoFile);
};
this.registerVideoFrameCallback();
}
getAudioTrack() {
if ((this.video as any).captureStream) {
// videoAudioTrack: MediaStreamTrack
this.videoAudioTrack = (this.video as any).captureStream().getAudioTracks()[0];
} else if ((this.video as any).mozCaptureStream) {
this.videoAudioTrack = (this.video as any).mozCaptureStream().getAudioTracks()[0];
} else {
const destination = this.audioContext.createMediaStreamDestination();
this.sourceNode.connect(destination);
this.videoAudioTrack = destination.stream.getAudioTracks()[0];
}
return this.videoAudioTrack;
}
Other TS keeping relevant code:
initVideoElements() {
this.video = this.videoElementRef.nativeElement;
this.video.style.display = 'none';
this.videoCanvas = this.videoCanvasElementRef.nativeElement;
this.videoCanvas.style.display = 'none';
this.videoCanvasCtx = this.videoCanvas.getContext('2d');
this.segmentationMaskCanvas = this.segmentationMaskCanvasElementRef.nativeElement;
this.segmentationMaskCanvasCtx = this.segmentationMaskCanvas.getContext('2d');
this.segmentationMaskCanvas.style.display = 'none';
}
onVideoFileSelected(file: File) {
this.videoFile = file; // videoFile: File;
}
async startVideoProcessing() {
// @ViewChild('uploadedVideo', { static: false }) videoElementRef: ElementRef<HTMLVideoElement>;
// video: HTMLVideoElement;
this.video.src = URL.createObjectURL(this.videoFile);
this.silenceVideoAudio();
this.video.onloadeddata = async () => {
await this.videoProcessing();
};
this.video.onended = async () => {
this.recorder.stop()
};
}
// This method connects the AudioContext and puts the gain at 0 since I hide the videos with CSS
silenceVideoAudio() {
this.audioContext = new AudioContext();
this.sourceNode = this.audioContext.createMediaElementSource(this.video);
const gainNode = this.audioContext.createGain();
gainNode.gain.value = 0;
this.sourceNode.connect(gainNode);
gainNode.connect(this.audioContext.destination);
}
registerVideoFrameCallback(): void {
const doSomethingWithTheFrame = (now: number, metadata: any) => {
this.processingProgress = parseFloat(metadata.mediaTime.toFixed(2)) / this.video.duration;
this.cdRef.detectChanges();
this.imageSegmenter.segmentForVideo(this.video, metadata.presentedFrames, async segmentationMask => await this.segmentationVideoCallback(segmentationMask));
this.videoElementRef.nativeElement.requestVideoFrameCallback(doSomethingWithTheFrame);
};
this.video.currentTime = 0;
this.videoElementRef.nativeElement.requestVideoFrameCallback(doSomethingWithTheFrame);
}
async segmentationVideoCallback(segmentationMask){
// Callback function that segments the video onto the canvas that I capture the stream of
await this.drawSegmentationResult(segmentationMask.categoryMask);
}
When I check consoles it seems the MediaRecorder is going inactive by the time the video onended is hit. When it goes inactive it seems it triggers the ondataavailable and onstop 100% of the time on desktop, but on mobile with audio it only happens some times. When it does happen I have all of the data necessary.
One thing I tried was requestData() on every requestVideoFrameCallback but the ondataavailable listener was equally unreliable. I would see it get into a conditional to call it, and the recorder was recording but the event listener would not be triggered.
Other things I tried were to wrap various parts in runOutsideAngular to make sure it wasn't tripped up by the zone, adding a timeslice in start(), using addEventListener, attempting to stop all tracks previous to stop(), and more. None of this consistently worked for me on mobile when an AudiContext was attached.
There are no errors logged with onerror but this could just also not be firing when it should?
One thing I noticed as well is that sometimes it would not trigger, but when I close my lock screen or other similar events and then come back, it has triggered the event (maybe as a result of something that was hanging on being canceled even when inactive?)
Can anyone help me figure out how to consistently end up with a file that is made up of the altered video canvas stream and the audio from the original video? Either using the MediaRecorder as I am here or otherwise?
If you are experiencing any of these symptoms with your MediaRecorder on mobile devices:
0 / MediaRecorder outputting 0 bytesMediaRecorderMediaRecorder finishingMediaRecorder not startingThen I would suggest you do a few things to correct the issue:
MediaRecorder has quite a few rules on when it goes inactive and when it will output data. Here is some information on what exactly those conditions are with special note of the start(), stop(), and state methods/properties: https://www.w3.org/TR/mediastream-recording/#mediarecorder-methodsThe answer to this problem was mostly related to check 2 after refactoring mobile for check 1. This issue was being caused by the order of events & async code & what I believe was effectively a race condition.
Here is the final code that I used to correct the issue above. The biggest changes were that I broke out a lot of the logic into smaller methods and made sure they executed fully, removed autoplay because of the tighter mobile restrictions, used this.video.load() after setting the video.src, awaited this.video.play(), moved my this.recorder.start() to begin AFTER starting the requestVideoFrameCallback processing.
HTML:
<ion-button (click)="startProcessingA()">Process Video</ion-button>
<video id="uploadedVideo" #uploadedVideo webkit-playsinline playsinline></video>
<canvas id="canvasVideo" #canvasVideo></canvas>
<canvas id="segmentationMaskCanvasVideo" #segmentationMaskCanvasVideo></canvas>
TS:
initVideoElements() {
this.video = this.videoElementRef.nativeElement;
this.video.style.display = 'none';
this.videoCanvas = this.videoCanvasElementRef.nativeElement;
this.videoCanvas.style.display = 'none';
this.videoCanvasCtx = this.videoCanvas.getContext('2d');
this.segmentationMaskCanvas = this.segmentationMaskCanvasElementRef.nativeElement;
this.segmentationMaskCanvasCtx = this.segmentationMaskCanvas.getContext('2d');
this.segmentationMaskCanvas.style.display = 'none';
}
async onVideoFileSelected(file: File) {
this.video.src = URL.createObjectURL(file);
this.video.onended = async () => {
this.recorder.stop();
URL.revokeObjectURL(this.video.src);
};
this.video.load();
if (this.isDesktop) {
this.startProcessingA();
}
}
async startProcessingA() {
await this.silenceVideoAudio();
await this.setupStreamRecorder();
await this.setupImageSegmenter();
await this.video.play();
this.registerVideoFrameCallback();
}
async silenceVideoAudio() {
return new Promise((resolve, reject) => {
this.audioContext = new AudioContext();
this.sourceNode = this.audioContext.createMediaElementSource(this.video);
const gainNode = this.audioContext.createGain();
gainNode.gain.value = 0;
this.sourceNode.connect(gainNode);
gainNode.connect(this.audioContext.destination);
resolve(null);
});
}
registerVideoFrameCallback(): void {
const doSomethingWithTheFrame = (now: number, metadata: any) => {
if (this.recorder.state !== 'recording') {
this.recorder.start();
}
this.imageSegmenter.segmentForVideo(this.video, metadata.presentedFrames, async segmentationMask => await this.segmentationVideoCallback(segmentationMask));
this.video.requestVideoFrameCallback(doSomethingWithTheFrameNotFirefox);
};
this.video.currentTime = 0;
this.video.requestVideoFrameCallback(doSomethingWithTheFrameNotFirefox);
}
async setupStreamRecorder() {
return new Promise((resolve, reject) => {
this.stream = new MediaStream();
this.stream.addTrack(this.getAudioTrack());
this.stream.addTrack(this.videoCanvas.captureStream(30).getVideoTracks()[0]);
const options = {
mimeType: this.mimeType
};
this.recorder = new MediaRecorder(this.stream, options);
this.recorder.ondataavailable = (event) => {
if (event.data && event.data.size > 0) {
this.chunks.push(event.data);
}
};
this.recorder.onerror = (event) => {
console.error(event);
throw new Error(event.toString());
};
this.recorder.onstop = () => {
const videoBlob = new Blob(this.chunks, { type: this.mimeType });
const videoFile = new File([videoBlob], this.mimeType === 'video/mp4' ? 'video-name.mp4' : 'video-name.webm', { type: this.mimeType });
};
resolve(null);
});
}
getAudioTrack() {
const destination = this.audioContext.createMediaStreamDestination();
this.sourceNode.connect(destination);
this.videoAudioTrack = destination.stream.getAudioTracks()[0];
return this.videoAudioTrack;
}