I am currently facing an issue while attempting to decode H264 frames using the appsrc element in GStreamer. I have constructed the following pipeline within my application:
appsrc name=mysource ! video/x-h264, alignment=au, stream-format=byte-stream ! h264parse ! video/x-h264, alignment=au, stream-format=byte-stream ! omxh264dec ! video/x-raw, format=NV16_10LE32, width=1920, height=1080, framerate=30/1 ! filesink location=ball.raw
To feed the appsrc with frames, I am reading an H264 file frame by frame since I store each frame in a structure along with its size. Then, I push the frames into the appsrc using the following function:
void VDRE2GStreamerDecodingDisplaySinkTask::PushBufferToGStreamer(uint8_t* encoded_frame, uint32_t frame_size)
{
GstFlowReturn ret;
GstMapInfo map;
GstBuffer *buffer = gst_buffer_new_and_alloc(frame_size);
gst_buffer_map(buffer, &map, GST_MAP_WRITE);
map.data = encoded_frame;
gst_buffer_unmap(buffer, &map);
/* Push the buffer into the appsrc */
g_signal_emit_by_name(m_appsrc, "push-buffer", buffer, &ret);
/* Free the buffer now that we are done with it */
gst_buffer_unref(buffer);
if (ret != GST_FLOW_OK) {
g_print("PushBufferToGStreamer: Error!\n");
}
}
However, when I execute the pipeline and set GST_DEBUG=4, I encounter the following error messages:
Decode Thread: Frame to Process : 1 :s 246
Decode Thread: Frame to Process : 2 :s 232
Decode Thread: Frame to Process : 3 :s 854
Decode Thread: Frame to Process : 4 :s 952
Decode Thread: Frame to Process : 5 :s 350
Decode Thread: Frame to Process : 6 :s 1038
Decode Thread: Frame to Process : 7 :s 1447
Decode Thread: Frame to Process : 8 :s 1316
Decode Thread: Frame to Process : 9 :s 1297
Decode Thread: Frame to Process : 10 :s 1205
0:00:02.370465675 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
0:00:02.370549896 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 1140 will be dropped
Decode Thread: Frame to Process : 11 :s 1222
Decode Thread: Frame to Process : 12 :s 1189
Decode Thread: Frame to Process : 13 :s 1094
Decode Thread: Frame to Process : 14 :s 930
Decode Thread: Frame to Process : 15 :s 1000
Decode Thread: Frame to Process : 16 :s 888
Decode Thread: Frame to Process : 17 :s 901
Decode Thread: Frame to Process : 18 :s 788
Decode Thread: Frame to Process : 19 :s 833
Decode Thread: Frame to Process : 20 :s 764
Decode Thread: Frame to Process : 21 :s 727
Decode Thread: Frame to Process : 22 :s 616
Decode Thread: Frame to Process : 23 :s 728
Decode Thread: Frame to Process : 24 :s 622
Decode Thread: Frame to Process : 25 :s 609
Decode Thread: Frame to Process : 26 :s 657
Decode Thread: Frame to Process : 27 :s 651
Decode Thread: Frame to Process : 28 :s 621
Decode Thread: Frame to Process : 29 :s 534
Decode Thread: Frame to Process : 30 :s 2989
0:00:03.033182378 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
0:00:03.033258709 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 117 will be dropped
0:00:03.033309439 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
0:00:03.033369800 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 1468 will be dropped
Decode Thread: Frame to Process : 31 :s 411
Decode Thread: Frame to Process : 32 :s 465
Decode Thread: Frame to Process : 33 :s 514
Decode Thread: Frame to Process : 34 :s 520
Decode Thread: Frame to Process : 35 :s 519
Decode Thread: Frame to Process : 36 :s 476
Decode Thread: Frame to Process : 37 :s 448
Decode Thread: Frame to Process : 38 :s 544
Decode Thread: Frame to Process : 39 :s 523
Decode Thread: Frame to Process : 40 :s 454
Decode Thread: Frame to Process : 41 :s 561
Decode Thread: Frame to Process : 42 :s 409
Decode Thread: Frame to Process : 43 :s 506
Decode Thread: Frame to Process : 44 :s 330
Decode Thread: Frame to Process : 45 :s 2180
0:00:03.530095934 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
0:00:03.530192075 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 117 will be dropped
0:00:03.530251666 654
So then I analyzed the encoded_data to check if there are valid NAL units:
have analyzed the encoded_frame data to determine if it contains valid NAL units. Here's a summary of your findings for the first five data sets:
Encoded data values in hexadecimal: 0, size: 3241
00 00 00 01 27 7A 00 28 B6 D0 0C E8 07 80 22 7E 27 01 6A 02 04 04 80 00 00 03 00 80 00 00 1E 19 A8 00 3D 09 00 01 6E 37 FF FE 05 00 00 00 01 28 EE 3C
Encoded data values in hexadecimal: 1, size: 246
00 00 00 01 21 E2 01 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 09 36 B5 84 05 B0 C9 8C DA C9 6F 9D F0 00
Encoded data values in hexadecimal: 2, size: 232
00 00 00 01 21 E4 02 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 16 D0 00 00 01 21 00 1F E3 90 08 21
Encoded data values in hexadecimal: 3, size: 854
00 00 00 01 21 E6 03 08 8B 7F 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 06 2C 00 00 01 21 00 1F E3 98 0C 22
Encoded data values in hexadecimal: 4, size: 952
00 00 00 01 21 E8 04 0E DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 97 80 00 00 01 21 00 1F E3 A0 10 3B
It seems that the analyzed data contains valid NAL units, as expected. I have attempted a similar test using a file source and launching the pipeline with gst-launch as follows:
gst-launch-1.0 -e filesrc location=ball2.h264 ! video/x-h264, alignment=au, width=1920, height=1080, framerate=30/1 ! h264parse ! omxh264dec ! video/x-raw, format=NV16_10LE32, width=1920, height=1080, framerate=30/1 ! filesink location=ball_gst.raw
During this test, I did not encounter any issues or error messages with GST_DEBUG=4. Additionally, the raw file was populated with bytes, unlike the appsrc version where the content always appears empty (presumably because it is not being decoded at any point).
I even tried with a identity dump=true element between filesrc and h264parse and I also have the same bytes I am injecting to appsrc So I don't know what can be the difference
I fixed it by removing the caps specification!!
If anyone is wondering how to do this pipeline. This is the final code to create it (Finally I add a fakesink to avoid the generation of big raw files)
m_pipeline = gst_pipeline_new("mypipeline");
m_appsrc = gst_element_factory_make("appsrc", "mysource");
GstElement *h264_parser = gst_element_factory_make("h264parse", "myparser");
GstElement *decoder = gst_element_factory_make("omxh264dec", "mydec");
GstElement *fakesink = gst_element_factory_make("fakesink", "mysink");
/* g_object_set(G_OBJECT (filesink), "location", "ball_inside.raw",
NULL); */
// Check if all elements were created
if (!m_pipeline && m_appsrc && h264_parser && decoder && fakesink)
{
g_print("Could not gst_element_factory_make, terminating\n");
return;
}
// Create gstreamer loop
loop = g_main_loop_new(NULL, FALSE);
// add a message handler
bus = gst_pipeline_get_bus(GST_PIPELINE(m_pipeline));
bus_watch_id = gst_bus_add_watch(bus, bus_call, loop);
gst_object_unref(bus);
// add all elements into pipeline
gst_bin_add_many(GST_BIN(m_pipeline), m_appsrc, h264_parser, decoder, fakesink, NULL);
// link elements into pipe: appsrc -> jpegenc -> appsink
gst_element_link(m_appsrc, h264_parser);
gst_element_link(h264_parser, decoder);
gst_element_link(decoder, fakesink);
GstPad *encoder_pad;
encoder_pad = gst_element_get_static_pad(decoder, "src");
gst_pad_add_probe(encoder_pad, GST_PAD_PROBE_TYPE_BUFFER, (GstPadProbeCallback)probe_source_callback, NULL, NULL);
encoder_pad = gst_element_get_static_pad(decoder, "sink");
gst_pad_add_probe(encoder_pad, GST_PAD_PROBE_TYPE_BUFFER, (GstPadProbeCallback)probe_sink_callback, NULL, NULL);
gst_object_unref (encoder_pad);
g_print("Setting g_main_loop_run to GST_STATE_PLAYING\n");
// Start pipeline so it could process incoming data
gst_element_set_state(m_pipeline, GST_STATE_PLAYING);
I also add those Pads to check that I have buffers on both sides of the decoder.