I detect new PES packet in PES demultiplexor searching packet_start_code_prefix (0x000001). When it occures then I can read PES_packet_length and so I can extract the current PES packet from byte stream. But if it is a H.264 video stream then PES_packet_length=0.
How to extract PES packet in such a case? 0x000001 also may occur in H.264 nal unit byte stream so I can't use this prefix for finding next PES packet.
I noticed that in every H.264 PES packet last nal unit in PES packet is a Filler data (nal_unit_type=12). Do I need to use this fact for detecting the end of current PES packet?
Normally no, this is impossible without knowing the length the the PES packet. However, Because you limit yourself to H.264, we can take advantage of a lucky accident.
An h.264 stream_id is 0xE0. The first bit of a nalu is always 0. so 000001E0 happens to be illegal within an annex B Stream. You must still parse the PES header to determine its length, because the first byte after the PES header may be the tail of a previous NALU, and thus may not necessarily be an annex b start code.
Keeping this for posterity. You can not simply look for start codes, you need to parse the packet. If this is a transport stream You find the start of the PES by looking for the payload unit start indicator. Then parse the adaption field if on exists. Now you will have your start code (000001E0 in this case. Then look look at the flags. Parse out the 33 bit PTS/DTS (you will need that for playback) and skip any optional fields (determined by the flags in the PES header). You now will have the start of your h.264 ES. Continue parsing the TS. for every TS with the same PID and payload unit start indicator = false, you are reading the frame. Once the payload unit start indicator is true, you have a new PES packer/frame.