javascriptpythonreactjsdjangowebsocket

Display encoded video frames using React and Django

I'm new to web development and have been trying to solve a problem for some time but no luck. I'm using React and Django

The thing is, there is a 3rd party application that performs some image processing using opencv on video frames and I've to display those encoded frames on the web browser.

I want to receive those frames using Django API, decode them, and display them using React JS, also returning a response with every frame to that 3rd party app.

I've prepared a flowchart of how things should work but haven't been able to start at all.

Flowchart:

Flowchart

The outcome on the browser should appear something like this.

Outcome:

enter image description here

Need to know how to approach this, shall I use WebSockets or can I send the encoded frames directly to React taking Django out of the picture.

Edit:

  1. The frames will be served by the 3rd party app in cv2.imencode('.jpg', frame) encoded format along with some other data in a JSON packet.
  2. The decoding needs to be done by Django or React (not sure which one will or should handle this)
  3. The frames will keep on updating as if a real-time video is playing, ie. the moment a new frame is received, it must replace the old frame. The frame rate will be around 25 fps.
  4. A response must be returned for each frame. Django needs to perform anything apart from serving frames and sending back a response.
  5. Since there will be multiple cameras to be displayed simultaneously, do I need to use more than one port in Django or one is going to be sufficient?
Solution

  • Creating a Django endpoint: WebSockets to receive encoded frames in Django, decode them, and return a response, while continuously updating the video frame on the web browser.

    import cv2
import json
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from channels.layers import get_channel_layer
from asgiref.sync import async_to_sync


@csrf_exempt
def process_frames(request):
    if request.method == 'POST':
        data = json.loads(request.body)
        
        # Extract the encoded frames and other data from the JSON packet
        encoded_frames = data['frames']
        # Process other data as needed
        
        # Decode the frames using cv2.imdecode()
        decoded_frames = []
        for encoded_frame in encoded_frames:
            frame = cv2.imdecode(encoded_frame, cv2.IMREAD_COLOR)
            decoded_frames.append(frame)
        
        # Perform any necessary operations with the frames
        
        # Return a response for each frame
        response = {'status': 'success'}
        return JsonResponse(response)

    For video streaming you can either use browser(HTML) for video rendering or React(JS) for video rendering. Both have its pros and cons.

    
<!DOCTYPE html>
<html>
<head>
    <title>Integrating inside HTML</title>
</head>
<body>
    <video id="videoPlayer" autoplay controls></video>

    <script>
        const video = document.getElementById('videoPlayer');

        function updateVideoFrame(frame) {
            const blob = new Blob([frame], { type: 'image/jpeg' });
            const frameURL = URL.createObjectURL(blob);
            video.src = frameURL;
        }

        // Make a request to the Django endpoint to receive the frames
        setInterval(() => {
            fetch('/process_frames', { method: 'POST' })
                .then(response => response.json())
                .then(data => {
                    if (data.status === 'success') {
                        updateVideoFrame(data.frame);
                    }
                })
                .catch(error => {
                    console.error('Error:', error);
                });
        }, 40); // Adjust the interval to achieve the desired frame rate (25 fps = 40 ms delay)
    </script>
</body>
</html>

    Integrating inside JS

    import React, { useEffect, useState } from 'react';

const VideoPlayer = () => {
    const [frame, setFrame] = useState(null);

    useEffect(() => {
        const fetchFrame = async () => {
            try {
                const response = await fetch('/process_frames', { method: 'POST' });
                const data = await response.json();

                if (data.status === 'success') {
                    setFrame(data.frame);
                }
            } catch (error) {
                console.error('Error:', error);
            }
        };

        // Fetch frames at the desired frame rate
        const intervalId = setInterval(fetchFrame, 40); // Adjust the interval to achieve the desired frame rate (25 fps = 40 ms delay)

        return () => {
            clearInterval(intervalId);
        };
    }, []);

    const videoSource = frame ? URL.createObjectURL(new Blob([frame], { type: 'image/jpeg' })) : '';

    return (
        <video src={videoSource} autoPlay controls />
    );
};

export default VideoPlayer;

    EDIT

    Django endpoint using Django Channels

    # This is a template code for using Django Channels 
import cv2
import json
from channels.generic.websocket import WebsocketConsumer


class FrameProcessingConsumer(WebsocketConsumer):
    def receive(self, text_data=None, bytes_data=None):
        if bytes_data:
            # Extract the encoded frames and other data from the JSON packet
            data = json.loads(bytes_data.decode())
            encoded_frames = data['frames']
            # Process other data as needed
            
            # Decode the frames using cv2.imdecode()
            decoded_frames = []
            for encoded_frame in encoded_frames:
                frame = cv2.imdecode(encoded_frame, cv2.IMREAD_COLOR)
                decoded_frames.append(frame)
            
            # Perform any necessary operations with the frames
            
            # Return a response for each frame
            response = {'status': 'success'}
            self.send(json.dumps(response))


@csrf_exempt
def process_frames(request):
    if request.method == 'POST':
        data = json.loads(request.body)
        
        # Extract the encoded frames and other data from the JSON packet
        encoded_frames = data['frames']
        # Process other data as needed
        
        # Decode the frames using cv2.imdecode()
        decoded_frames = []
        for encoded_frame in encoded_frames:
            frame = cv2.imdecode(encoded_frame, cv2.IMREAD_COLOR)
            decoded_frames.append(frame)
        
        # Perform any necessary operations with the frames
        
        # Return a response for each frame
        response = {'status': 'success'}
        return JsonResponse(response)

    Make changes as per your requirements.

    Hope this helps...