pythonopencvffmpegsubprocessvideo-streaming

Display stream with FFmpeg, python and opencv

Situation : I have a basler camera connected to a raspberry pi, and I am trying to livestream it's feed with FFmpg to a tcp port in my windows PC in order to monitor whats happening in front of the camera.

Things that work : I manage to set up a python script on the raspberry pi which is responsible for recording the frames, feed them to a pipe and streaming them to a tcp port. From that port, I am able to display the stream using FFplay.

My problem : FFplay is great for testing out quickly and easily if the direction you are heading is correct, but I want to "read" every frame from the stream, do some processing and then displaying the stream with opencv. That, I am not able to do yet.

Minimaly reprsented, that's the code I use on the raspberry pi side of things :

command = ['ffmpeg',
           '-y',
           '-i', '-',
           '-an',
           '-c:v', 'mpeg4',
           '-r', '50',
           '-f', 'rtsp',
           '-rtsp_transport',
           'tcp','rtsp://192.168.1.xxxx:5555/live.sdp']

p = subprocess.Popen(command, stdin=subprocess.PIPE) 

while camera.IsGrabbing():  # send images as stream until Ctrl-C
    grabResult = camera.RetrieveResult(100, pylon.TimeoutHandling_ThrowException)
    
    if grabResult.GrabSucceeded():
        image = grabResult.Array
        image = resize_compress(image)
        p.stdin.write(image)
    grabResult.Release()

On my PC if I use the following FFplay command on a terminal, it works and it displays the stream in real time :

ffplay -rtsp_flags listen rtsp://192.168.1.xxxx:5555/live.sdp?tcp

On my PC if I use the following python script, the stream begins, but it fails in the cv2.imshow function because I am not sure how to decode it:

import subprocess
import cv2

command = ['C:/ffmpeg/bin/ffmpeg.exe',
           '-rtsp_flags', 'listen',
           '-i', 'rtsp://192.168.1.xxxx:5555/live.sdp?tcp?', 
           '-']

p1 = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE)

while True:
    frame = p1.stdout.read()
    cv2.imshow('image', frame)
    cv2.waitKey(1)

Does anyone knows what I need to change in either of those scripts in order to get i to work?

Thank you in advance for any tips.

Solution

  • We may read the decoded frames from p1.stdout, convert it to NumPy array, and reshape it.

    Now we can show the frame by calling cv2.imshow('image', frame).

    The solution assumes, we know the video frame size (width and height) from advance.

    The code sample below, includes a part that reads width and height using cv2.VideoCapture, but I am not sure if it's going to work in your case (due to '-rtsp_flags', 'listen'. (If it does work, you can try capturing using OpenCV instead of FFmpeg).

    The following code is a complete "working sample" that uses public RTSP Stream for testing:

    import cv2
import numpy as np
import subprocess

# Use public RTSP Stream for testing
in_stream = 'rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4'

if False:
    # Read video width, height and framerate using OpenCV (use it if you don't know the size of the video frames).

    # Use public RTSP Streaming for testing:
    cap = cv2.VideoCapture(in_stream)

    framerate = cap.get(5) #frame rate

    # Get resolution of input video
    width  = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    # Release VideoCapture - it was used just for getting video resolution
    cap.release()
else:
    # Set the size here, if video frame size is known
    width = 240
    height = 160


command = ['C:/ffmpeg/bin/ffmpeg.exe', # Using absolute path for example (in Linux replacing 'C:/ffmpeg/bin/ffmpeg.exe' with 'ffmpeg' supposes to work).
           #'-rtsp_flags', 'listen',   # The "listening" feature is not working (probably because the stream is from the web)
           '-rtsp_transport', 'tcp',   # Force TCP (for testing)
           '-max_delay', '30000000',   # 30 seconds (sometimes needed because the stream is from the web).
           '-i', in_stream,
           '-f', 'rawvideo',           # Video format is raw video
           '-pix_fmt', 'bgr24',        # bgr24 pixel format matches OpenCV default pixels format.
           '-an', 'pipe:']

# Open sub-process that gets in_stream as input and uses stdout as an output PIPE.
ffmpeg_process = subprocess.Popen(command, stdout=subprocess.PIPE)

while True:
    # Read width*height*3 bytes from stdout (1 frame)
    raw_frame = ffmpeg_process.stdout.read(width*height*3)

    if len(raw_frame) != (width*height*3):
        print('Error reading frame!!!')  # Break the loop in case of an error (too few bytes were read).
        break

    # Convert the bytes read into a NumPy array, and reshape it to video frame dimensions
    frame = np.frombuffer(raw_frame, np.uint8).reshape((height, width, 3))

    # Show the video frame
    cv2.imshow('image', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
  

ffmpeg_process.stdout.close()  # Closing stdout terminates FFmpeg sub-process.
ffmpeg_process.wait()  # Wait for FFmpeg sub-process to finish

cv2.destroyAllWindows()

    Sample frame (just for fun):
    enter image description here

    Update:

    Reading width and height using FFprobe:

    When we don't know the video resolution from advance, we may use FFprobe for getting the information.

    Here is a code sample for reading width and height using FFprobe:

    import subprocess
import json

# Use public RTSP Stream for testing
in_stream = 'rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4'

probe_command = ['C:/ffmpeg/bin/ffprobe.exe',
                 '-loglevel', 'error',
                 '-rtsp_transport', 'tcp',  # Force TCP (for testing)]
                 '-select_streams', 'v:0',  # Select only video stream 0.
                 '-show_entries', 'stream=width,height', # Select only width and height entries
                 '-of', 'json', # Get output in JSON format
                 in_stream]

# Read video width, height using FFprobe:
p0 = subprocess.Popen(probe_command, stdout=subprocess.PIPE)
probe_str = p0.communicate()[0] # Reading content of p0.stdout (output of FFprobe) as string
p0.wait()
probe_dct = json.loads(probe_str) # Convert string from JSON format to dictonary.

# Get width and height from the dictonary
width = probe_dct['streams'][0]['width']
height = probe_dct['streams'][0]['height']