I am currently working on a project where I control a robot arm in VR. I am doing this using A-Frame and an Oculus Quest 1.
To allow the user to see what they are doing with the robot arm, I want to stream the video of a webcam pointed at the arm to the Web-VR application. The webcam and headset are both on the same LAN, so I don't need the video to be accessible from the Internet.
What have I tried for streaming the webcam so far:
I have tried to watch all the OBS streams in VLC, the only one that seemed to work was the SRT stream with mode=listener. But when I tried to view that stream in A-Frame it didn’t work.
And now OBS does not let me stream using SRT anymore (I don’t know why). I will try to reinstall OBS to see if that helps.
What I have tried to view the stream in A-Frame:
<a-video src="protocol://145.89.161.93:8087" width="16" height="9" position="0 0 -10"></a-video>
<a-assets>
<video id="#webcam_stream" src="protocol://145.89.161.93:8087" playsinline> </video>
</a-assets>
<a-video src="#webcam_stream" width="16" height="9" position="0 0 -10"></a-video>
The HTTP stream does not seem to work as the server the Oculus connects with is run using HTTPS. As the headset otherwise will not allow usage of VR. With the other streams A-Frame doesn't seem to give an error on the connection or unknown source file, etc. But it does not display the stream/video either.
The other VLC streams come up with a cross-origin error, but that disappears when I use crossorigin="anonymous". But the a-video component can not find the #webcam_stream asset if I do this.
I think this tutorial would have helped a bit, even tough it’s not A-Frame. But the HTML code is not visible so it’s not very useful.
I do not very well know the subject of web development and streaming, but this is quite the essential part of our project.
Is there a solution to have a remote webcam stream visible in A-Frame, pre-existing code or a general direction/suggestion to look into?
All right, after lots of looking I have found a solution to my problem! There were multiple problems I had to tackle:
For the streaming protocol I ended up using HLS. It is not supported by most browsers and HTML5 by default. But with usage of HLS.js, we can add this support very easily.
Using this protocol, it also takes care of our security problems. As HLS streams are basically just short segments of video supplied using an HTTP/HTTPS server. Because our server is a simple https server, we only need to output the HLS files in a directory inside the server to make them available to the application.
If your stream is not on the same server as your application you will need to add CORS Headers to your HLS Host for the GET request. This is a requirement by the HLS.js library
To apply the stream on an entity in A-Frame, we need a <video> asset. After this asset has been loaded in, we can use a script to apply the stream on the asset. We can then use this asset as a source for entities and textures.
The code below does exactly this. As the entities are loaded in after the assets, adding a component to the <a-video> entity will invoke the script after the assets have been loaded in. We can then use the asset as a source for our <a-video> entity.
<!DOCTYPE html>
<html>
<head>
<script src="https://aframe.io/releases/1.2.0/aframe.min.js"></script>
<script src="hls.js"></script>
<script>
AFRAME.registerComponent('hls_stream', {
init: function(){
var video = document.getElementById('webcam_stream');
if(Hls.isSupported()) {
var hls = new Hls();
hls.loadSource('https://ip:port/path_to_hls_file/filename.m3u8');
hls.attachMedia(video);
hls.on(Hls.Events.MANIFEST_PARSED,function() {
video.play();
});
}
else if (video.canPlayType('application/vnd.apple.mpegurl')) {
video.src = 'https://ip:port/path_to_hls_file/filename.m3u8';
video.addEventListener('loadedmetadata',function() {
video.play();
});
}
}
})
</script>
</head>
<body>
<a-scene cursor="rayOrigin:mouse">
<a-assets>
<video id="webcam_stream" crossorigin="anonymous"></video>
</a-assets>
<a-video src="#webcam_stream" width="16" height="9" position="0 4.5 -20" hls_stream></a-video>
<a-entity id="camera" wasd-controls camera look-controls></a-entity>
<a-sky color="#111"></a-sky>
</a-scene>
</body>
</html>
Currently I am creating the HLS stream using OBS. You can find how to stream HLS using OBS here. OBS is relatively easy to set up, and allows for more then just a webcam, as you can capture almost any video, screen or other input source using OBS.
The only thing I am still struggling with is the delay in the stream of OBS. Because of the two second segments and because the length of the segment list of the HLS stream of OBS is 4, we usually get a delay of about 8 seconds. This is still larger than what I want, but it will do for now as starting a low delay HLS stream is a topic for another question.
Most browsers have autoplay for video files disabled by default. So you will need to manually start the video.play() command if auto play is disabled. Or you can enable autoplay in the settings of your browser and the video will video will automatically start.