I am using libstagefright to decode a 1020p video having baseline high and level 31 on android.
On emulator the video decoding fails, and i assume the reason is that softavc does not support high baseline according to the code here
But I tested this code on a real device which uses the OMX.MTK.VIDEO.DECODER.AVC decoder. And according to this link, this decoder supports high baseline decoding with level 31.
But the video result is garbled.
Does anyone have any insight, why this is so? And what could be the possible solution?
There could be 3 reasons for garbled output
The decoder employs a stride which you haven't factored into your calculations. Recommended Solution: Please check OMXcomponent's port parameters and look for stride and make suitable modifications.
I presume you are decoding 1080, please consider 1088 for calculations instead of 1080. This can be confirmed if your output has clear luma, but jumbled chroma. Does 720p play fine for you?
If it's neither of the above 2 conditions are true, then the decoder may be outputting a tiled output. Please check with the vendor's specifications. If this is true, you will need to convert from the tiled format to a more common format like NV12.
EDIT: I think tiling is your problem. Please check this github commit which is related to your problem and has a solution for color conversion.