I have setup an RTSP stream using ffmpeg:
ffmpeg -loglevel warning -re -stream_loop -1 -i path/to/file.mp4 -c:v libx264 -preset superfast -tune zerolatency -pix_fmt yuv420p -b:v 600k -max_muxing_queue_size 1024 -profile:v baseline -an -filter:v scale=1280:-1 -f rtsp rtsp://<IP>:8554/test
Then I tried to access that stream with the following gstreamer pipeline:
gst-launch-1.0 rtspsrc location=rtsp://<IP>:8554/test drop-on-latency=true ! rtph264depay ! h264parse ! queue ! nvv4l2decoder ! queue ! nvvidconv ! 'video/x-raw, format=(string)RGBA' ! fakesink
I used a fakesink just to see if the pipeline worked and if the NVDEC was enabled (I am using a Jetson Orin Nano to run this pipeline). However, when running this pipeline I get the following error:
libEGL warning: DRI3: failed to query the version
libEGL warning: DRI2: failed to authenticate
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.12.13:8554/flare
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Stream format not found, dropping the frame
Stream format not found, dropping the frame
As you can see I get several messages of the type Stream format not found, dropping the frame
In order to debug I changed the RTSP source, and created an RTSP server using: (compiled the test-launch-c:
./test-launch "filesrc location=video.m4v ! qtdemux ! rtph264pay name=pay0 pt=96 "
The video.m4v was H264 encoded.
Now, running the previous pipeline with the new RTSP source I have no error.
Why does it happen and how can I change my failing gstreamer pipeline to account for different source formats?
As advised by SeB I run the previous pipeline in debug mode for both situations, here are my findings:
FFMPEG Stream (The problematic one)
/GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
RTSP Stream (Working fine)
/GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)596, height=(int)336, framerate=(fraction)30000/1001, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)high, level=(string)3
I saw that the profile and level in the working pipeline are "high" and "3" respectively, so I tried to change my previous pipeline to:
gst-launch-1.0 rtspsrc location=rtsp://192.168.18.10:8554/test drop-on-latency=true ! rtph264depay ! h264parse ! video/x-h264,profile=high,level=3 ! queue ! nvv4l2decoder enable-max-performance=1 ! queue ! nvvidconv ! 'video/x-raw, format=(string)RGBA' ! fakesink
Although I do not see the error anymore:
libEGL warning: DRI3: failed to query the version
libEGL warning: DRI2: failed to authenticate
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.18.10:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
The decoder is not being used.
Do I need to add more caps or change where they are applied?
You may adjust your ffmpeg command to use H264 high profile with level 3.1:
... -profile:v high -level:v 3.1 ...
Also see: https://en.wikipedia.org/wiki/Advanced_Video_Coding
Switching to gstreamer RTSP server may also be a better option, especially if later moving to a higher grade Orin module, this may leverage HW NVENC.