I am writing a TS packetizer in C and I am having trouble packing AVCC video. The live stream comes from the source and its raw payload is sent to the TS packetizer. When the stream just starts, the packetizer first writes PAT and PMT, then starts packing the payload, divided into 188-byte packets - [Header, Adaptation field, data], [Header, data], [Header, data] ... [Header, Adaptation field, data]. The packetized stream is output to a file. When I pack a H264 AnnexB stream, VLC plays this file fine, but it cannot play h264 AVCC.
So what is the difference in packing AVCC from AnnexB? What am I missing? What additional steps does AVCC require?
Mapping AVC into MPEG-2 Transport Stream requires AnnexB (start code separated NAL units) - per specification. There is no way to map size prefixed NAL units (as in MP4) into a MPEG-2 Transport Stream. You can convert MP4 style AVC into Annex B and then put into a MPEG-2 Transport stream. Although the mapping of AVC into MPEG-2 Transport Stream is a somewhat mechanical process. It is easy to under estimate the complexity. You have to create PES headers with the correct DTS/PTS, video descriptor, correct PCR and more. Have a look at ISO/IEC 13818-1:2013 and related specifications. Also have a look at existing open source implementations to understand the magnitude of the task.
VLC is a very lenient player - so that a file plays in VLC - says very little about the file's correctness.