I am having trouble setting up a GStreamer pipeline to forward a video stream over UDP via OpenCV. I have a laptop, and an AGX Xavier connected to the same network. The idea is to forward the webcam video feed to AGX which will do some OpenCV optical flow estimation on the GPU (in Python), draw flow vectors on the original image and send it back to my laptop. Up until now, I can configure two pipelines. As a minimum example, I have made two bash scripts and a Python script that ideally would function as pass-through over OpenCV's VideoCapture and VideoWriter objects.
servevideo.bash:
#!/bin/bash
gst-launch-1.0 v4l2src device=[device-fd] \
! video/x-raw, width=800, height=600, framerate=24/1 \
! jpegenc ! rtpjpegpay ! rtpstreampay \
! udpsink host=[destination-ip] port=12345
receivevideo.bash:
#!/bin/bash
gst-launch-1.0 -e udpsrc port=12344 \
! application/x-rtp-stream,encoding-name=JPEG \
! rtpstreamdepay ! rtpjpegdepay ! jpegdec \
! autovideosink
If I run these two scripts on either the same computer or on two different computers on the network, it works fine. When I throw my Python script (listed below) in the mix, I start to experience issues. Ideally, I would run the bash scripts on my laptop with the intended setup in mind while running the Python script on my Jetson. I would then expect to see the webcam video feed at my laptop after taking a detour around the Jetson.
webcam_passthrough.py:
#!/usr/bin/python3.6
import cv2
video_in = cv2.VideoCapture("udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink", cv2.CAP_GSTREAMER)
video_out = cv2.VideoWriter("appsrc ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[destination-ip] port=12344", cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)
while True:
ret, frame = video_in.read()
if not ret: break
video_out.write(frame)
cv2.imshow('Original', frame)
key = cv2.waitKey(1) & 0xff
if key == 27: break
cv2.destroyAllWindows()
video_out.release()
video_in.release()
With the following Python script, I can visualise the frames via cv2.imshow
received from the pipeline set up by the servevideo.bash
script. So I think my problem is connected to how I am setting up the VideoWriter video_out
in OpenCV. I have verified my two bash scripts are working when I am relaying the webcam video feed between those two pipelines created, and I have verified that the cv2.VideoCapture
receives the frames. I am no expert here, and my GStreamer knowledge is almost non-existent, so there might be several misunderstandings in my minimum example. It would be greatly appreciated if some of you could point out what I am missing here.
I will also happily provide more information if something is unclear or missing.
EDIT: So it seems the intention of my minimum example was not clearly communicated.
The three scripts provided as a minimum example serve to relay my webcam video feed from my laptop to the Jetson AGX Xavier who then relays the video-feed back to the laptop. The servevideo.bash
creates a GStreamer pipeline on the laptop that uses v4l2 to grab frames from the camera and relay it on to a UDP socket. The webcam_passthrough.py
runs on the Jetson where it "connects" to the UDP socket created by the pipeline running on the laptop. The Python script serves a passthrough which ideally will open a new UDP socket on another port and relay the frames back to the laptop. The receivevideo.bash
creates yet another pipeline on the laptop for receiving the frames that were passed through the Python script at the Jetson. The second pipeline on the laptop is only utilised for visualisation purpose. Ideally, this minimum example shows the "raw" video feed from the camera connected to the laptop.
The two bash scripts are working in isolation, both running locally on the laptop and running receivevideo.bash
remotely on another computer.
The cv2.VideoCapture
configuration in the Python script also seems to work as I can visualise the frames (with cv2.imshow
) received over the UDP socket provided by the servevideo.bash
script. This is working locally and remotely as well. The part that is causing me some headache (I believe) is the configuration of cv2.VideoWriter
; ideally, that should open a UDP socket which I can "connect" to via my receivevideo.bash
script. I have tested this locally and remotely but to no avail.
When I run receivevideo.bash
to connect to the UDP socket provided by the Python script I get the following output:
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
This does not seem wrong to me, I have tried to run the different scripts with GST_DEBUG=3 which gave some warnings, but as the pipeline configurations are basically the same in the bash scripts and for the cv2 VideoCapture
and VideoWriter
I do not add much value to those warnings. As an example I have included one such warning below:
0:00:06.595120595 8962 0x25b8cf0 WARN rtpjpegpay gstrtpjpegpay.c:596:gst_rtp_jpeg_pay_read_sof:<rtpjpegpay0> warning: Invalid component
This warning is printed continuously running the Python script with GST_DEBUG=3
. Running the receivevideo.bash
with the same debug level gave:
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
0:00:00.013911480 9078 0x55be0899de80 FIXME videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out:<jpegdec0> Sub-class should implement drain()
Setting pipeline to PLAYING ...
New clock: GstSystemClock
I hope my intention is clearer now, and as I already pointed out I believe something is wrong with my cv2.VideoWriter
in the Python script, but I am no expert and GStreamer is far from something that I use every day. Thus, I may have misunderstood something.
EDIT 2: So now I have tried to split the two pipelines into two separate processes as suggested by @abysslover. I still see the same result, and I still have no clue why that is. My current implementation of the Python script is listed below.
webcam_passthrough.py:
#!/usr/bin/python3.6
import signal, cv2
from multiprocessing import Process, Pipe
is_running = True
def signal_handler(sig, frame):
global is_running
print("Program was interrupted - terminating ...")
is_running = False
def produce(pipe):
global is_running
video_in = cv2.VideoCapture("udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink", cv2.CAP_GSTREAMER)
while is_running:
ret, frame = video_in.read()
if not ret: break
print("Receiving frame ...")
pipe.send(frame)
video_in.release()
if __name__ == "__main__":
consumer_pipe, producer_pipe = Pipe()
signal.signal(signal.SIGINT, signal_handler)
producer = Process(target=produce, args=(producer_pipe,))
video_out = cv2.VideoWriter("appsrc ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[destination-ip] port=12344", cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)
producer.start()
while is_running:
frame = consumer_pipe.recv()
video_out.write(frame)
print("Sending frame ...")
video_out.release()
producer.join()
The pipe that I have created between the two processes is providing a new frame as expected. When I try to listen to UDP port 12344 with netcat
, I do not receive anything that is the same thing as before. I also have a hard time understanding how differentiating the pipelines are changing much as I would expect them to already run in different contexts. Still, I could be wrong concerning this assumption.
you were very close to the solution. The problem lies in the warning you yourself noticed warning: Invalid component
. The problem is that rtp jpeg payloader gets stuck due to not supporting video format it is getting. Check this
However I was blind and missed what you wrote and went full debug mode into the problem.
So lets just keep the debug how-to for others or for similar problems:
1, First debugging step - check with wireshark if the receiving machine is getting udp packets on port 12344. Nope it does not.
2, Would this work without opencv stuff? Lets check with replacing opencv logic with some random processing - say rotation of video. Also eliminate appsrc/appsink to simplify.
Then I used this:
GST_DEBUG=3 gst-launch-1.0 udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! rotate angle=0.45 ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! queue ! udpsink host=[my ip] port=12344
Hm now I get weird warnings like:
0:00:00.174424533 90722 0x55cb38841060 WARN rtpjpegpay gstrtpjpegpay.c:596:gst_rtp_jpeg_pay_read_sof:<rtpjpegpay0> warning: Invalid component
WARNING: from element /GstPipeline:pipeline0/GstRtpJPEGPay:rtpjpegpay0: Invalid component
3, Quick search yielded above mentioned GStreamer forum page.
4, When I added video/x-raw,format=I420
after videoconvert it started working and my second machine started getting the udp packets.
5, So the solution to your problem is just limit the jpegenc to specific video format that the subsequent rtp payloader can handle:
#!/usr/bin/python3
import signal, cv2
from multiprocessing import Process, Pipe
is_running = True
def signal_handler(sig, frame):
global is_running
print("Program was interrupted - terminating ...")
is_running = False
def produce(pipe):
global is_running
video_in = cv2.VideoCapture("udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink", cv2.CAP_GSTREAMER)
while is_running:
ret, frame = video_in.read()
if not ret: break
print("Receiving frame ...")
pipe.send(frame)
video_in.release()
if __name__ == "__main__":
consumer_pipe, producer_pipe = Pipe()
signal.signal(signal.SIGINT, signal_handler)
producer = Process(target=produce, args=(producer_pipe,))
# the only edit is here, added video/x-raw capsfilter: <-------
video_out = cv2.VideoWriter("appsrc ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[receiver ip] port=12344", cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)
producer.start()
while is_running:
frame = consumer_pipe.recv()
rr = video_out.write(frame)
print("Sending frame ...")
print(rr)
video_out.release()
producer.join()