Decode frame from h264 stream to OpenCV Mat

Product: Bebop Product version: Software version v2.0.29 SDK version: ARSDK3_version_3_14_0 Use of libARController: Yes SDK platform: UNIX Reproductible with the official app: NO

I have been trying to convert an ARCONTROLLER_Frame_t from ARSDK 3 to an OpenCV image using Ubuntu 18.04, originally, the sample program provided made use of MPlayer to open a FIFO, where the program wrote all the frames it received from the drone, I managed to obtain images from said stream using ffmpeg while it was running. I also tried to give OpenCV said file to use as a source for VideoCapture, which worked but was horribly delayed. I am currently trying to give OpenCV frame by frame.

This is a screenshot of the definition of ARCONTROLLER_Frame_t, the documentation is very vague about how everything works.

I currently get the following image: Screen shot of the image obtained

I previously used this to try to decode it, assuming it was on an RGB format, this image reacts to interaction with the camera, so I assume it is the correct data:

void rawToMat(Mat &destImage, ARCONTROLLER_Frame_t &sourceImage) {
  if (sourceImage.used == 0) {
    return;
  }
  uchar *pointerImage = destImage.ptr(0);

  for (int i = 0; i < 480 * 856; i++) {
    pointerImage[3 * i] = sourceImage.data[3 * i + 2];
    pointerImage[3 * i + 1] = sourceImage.data[3 * i + 1];
    pointerImage[3 * i + 2] = sourceImage.data[3 * i];
  }
}

But I found out this provided the same output:

ARCONTROLLER_Frame_t newFrame = getCurrentFrame();
Mat currentImage = Mat(480, 856, CV_8UC3, newFrame.data);

Does someone know a way to get an actual image from this?

ARCONTROLLER_Frame_t does not contain an RGB image; it contains an H.264 frame that must be converted to an RGB image using an H.264 decoder (for example, see this response on the Parrot developer forum).

This old sample code from Parrot demonstrates how to decode frames using the ffmpeg/libav library (it's much too long to post here): https://github.com/Parrot-Developers/Samples/tree/59b6ba5cdc268fb6932d228db7b9169d9b69384c/Unix/BebopDroneDecodeStream