[SOLVED] gstreamer webrtc h264 playback stops after few frames in browser

gstreamer webrtc h264 playback stops after few frames in browser

I need help for debugging a probabilistic issue. I built a gstreamer pipeline to stream NVENC encoded h264 bitstreams(video only) to browser. Browser seldom plays properly. In most cases only few frames are rendered then the picture gets frozen.

The NVENC settings follow "https://cloud.google.com/solutions/gpu-accelerated-streaming-using-webrtc" which are h264 high profile & low latency high quality & NVENC_INFINITE_GOPLENGTH(some settings have been tried, like rateControlMode/enableVFR/sliceMode/repeatSPSPPS/outputAUD but no help). At runtime, NVENC encodes real-time rendered opengl fbo texture to h264 bitstreams and push them into gstreamer via appsrc. Currently the texture size is 512x512 and fed at 10/20/30 fps.

I use gstreamer 1.18.2, the pipeline is defined as "appsrc name=nvenc_src do-timestamp=1 ! video/x-h264, stream-format=byte-stream, alignment=au ! rtph264pay aggregate-mode=zero-latency ! queue ! application/x-rtp,media=video,encoding-name=H264,payload=123 ! webrtcbin bundle-policy=max-compat name=backend_webrtc".

The gstreamer part codes follow the sendrecv example(replacing libsoup with websocketpp and removing the recv logics).

The application is built as MSVC 2019 32-bit. The browser decoder is NVDEC. Exe application and js codes run on the same PC(windwos 10, gtx1060, driver version 460.89). I've tried in Chrome(87.0.4280.88) and edge(87.0.664.66). I also tried running the js codes in android(chrome) and ios(safari) and get the same results.

It can be concluded that NVENC generates 'correct' h264 bitstreams. I dump the raw h264 bitstreams into file. The file plays properly in VLC. I also tried pushing the dumped h264 bitstreams into gstreamer. The frozen issue still happens.

After the picture is frozen, the playback never recovers. the browser's 'webrtc-internals' shows that bytes/headerBytes/packests_Received keep growing, while frameReceived/framesDecoded/framesDropped stay unchanged.

Since the bitwise same h264 frames behave differently at different runs, I guess rtp timestamps might cause the issue. I've tried setting appsrc's do-timestamp to 0 and manually set gstbuffer's PTS but it does not help.

Solution

Here are few things that you need to pay attention to:

Infinite GOP will not work - you must configure NVENC to send a key frame every 30 - 60 frames.
Of course SPS-PPS NALs must come before each key frame.
Prohibit B-frames: WebRTC doesn't support them because they increase latency.
Startup codes between NALs must be 3-bytes startup codes: WebRTC doesn't respect 2-bytes startup codes. We bumped into this issue before and had to manually correct the startup codes.