androidc++arcoreyuvrgba

ARCore's ArFrame_acquireCameraImage method returns green image


I'm trying to get image from camera using ARCore.

I'm calling ArFrame_acquireCameraImage, which returns image with YUV_420_888 format. I also checked it using ArImage_getFormat method. It returns me 640x480 image. Then I obtain pixel stride for U plane to distinguish images with NV21 or YV12 format.

Then I combine Y, U, V arrays into single one using memcpy, encode it to Base64 (using function by J. Malinen) and print it to log.

Also I tried to perform YUV420p -> RGBA conversion using RenderScript Intrinsics Replacement Toolkit.

I have this code:

  LOGD("take frame");
  ArImage *image = nullptr;
  if (mArSession != nullptr && mArFrame != nullptr &&
      ArFrame_acquireCameraImage(mArSession, mArFrame, &image) == AR_SUCCESS) {
    const uint8_t *y;
    const uint8_t *u;
    const uint8_t *v;

    int planesCount = 0;
    ArImage_getNumberOfPlanes(mArSession, image, &planesCount);
    LOGD("%i", planesCount);

    int yLength, uLength, vLength;
    ArImage_getPlaneData(mArSession, image, 0, &y, &yLength);
    ArImage_getPlaneData(mArSession, image, 1, &u, &uLength);
    ArImage_getPlaneData(mArSession, image, 2, &v, &vLength);

    auto *yuv420 = new uint8_t[yLength + uLength + vLength];
    memcpy(yuv420, y, yLength);
    memcpy(yuv420 + yLength, u, uLength);
    memcpy(yuv420 + yLength + uLength, v, vLength);

    int width, height, stride;
    ArImage_getWidth(mArSession, image, &width);
    ArImage_getHeight(mArSession, image, &height);

    ArImage_getPlanePixelStride(mArSession, image, 1, &stride);

    //auto *argb8888 = new uint8_t[width * height * 4];

    renderscript::RenderScriptToolkit::YuvFormat format = renderscript::RenderScriptToolkit::YuvFormat::YV12;
    if(stride != 1) {
      format = renderscript::RenderScriptToolkit::YuvFormat::NV21;
    }
    LOGD("%i %i %i", width, height, format);
    
    /*renderscript::RenderScriptToolkit toolkit;
    toolkit.yuvToRgb(yuv420, argb8888, width, height, format);*/

    LOGD("%s", base64_encode(yuv420, yLength + uLength + vLength).c_str());

    // delete[](argb8888);
    delete[](yuv420);
  }
  if (image != nullptr) {
    ArImage_release(image);
  }

Full code in repo.

My phone is Xiaomi Mi A3. Also tried to run this on emulator, but it still gives me same picture.

Actual image should look like this:

actual image

However, my code prints this image (I decoded it using RAW Pixels):

Just a green image, but top pixels are lighter

Decoding parameters: NV21 preset with semi planar pixel plane

If I uncomment code for YUV420 -> ARGB conversion and print Base64 for argb8888 array, I will have this image: Black image with gray top pixels

Preset: RGB32, width: 640, height: 480. Base64 of this image.


Solution

  • I replaced RenderScript Intrinsics Replacement Toolkit (which have multithreading and SIMD) with code taken from TensorFlow. I see this advantages:

    1. It's simpler. Here's attempt to use RSIRT:
        auto *yuv420 = new uint8_t[yLength + uLength + vLength];
        memcpy(yuv420, y, yLength);
        memcpy(yuv420 + yLength, u, uLength);
        memcpy(yuv420 + yLength + uLength, v, vLength);
    
        renderscript::RenderScriptToolkit::YuvFormat format = 
        renderscript::RenderScriptToolkit::YuvFormat::YV12;
        if(stride != 1) {
          format = renderscript::RenderScriptToolkit::YuvFormat::NV21;
        }
    
        renderscript::RenderScriptToolkit toolkit;
        toolkit.yuvToRgb(yuv420, argb8888, width, height, format);
    

    It's line that I wrote to use TensorFlow code:

    ConvertYUV420ToARGB8888(y, u, v, argb8888, width, height, yStride, uvStride, uvPixelStride);
    

    As you see, RSIRT takes only planar image, while Tensorflow code is written to use image splitted by 3 planes, so you don't need to use memcpy. It's the reason why this decision won't hurt performance.

    1. I found out that raw image is big (1.2Mb), so I shouldn't use Base64 (I think that Logcat just cut my output, so I wasn't able to see image). Now I write image to app cache and take it using adb.

    Full code:

      ArImage *image = nullptr;
      if (mArSession != nullptr && mArFrame != nullptr &&
          ArFrame_acquireCameraImage(mArSession, mArFrame, &image) == AR_SUCCESS) {
        // It's image with Android YUV 420 format https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888
    
        const uint8_t *y;
        const uint8_t *u;
        const uint8_t *v;
    
        int planesCount = 0;
        ArImage_getNumberOfPlanes(mArSession, image, &planesCount);
        LOGD("%i", planesCount);
    
        int yLength, uLength, vLength, yStride, uvStride, uvPixelStride;
        ArImage_getPlaneData(mArSession, image, 0, &y, &yLength);
        ArImage_getPlaneData(mArSession, image, 1, &u, &uLength);
        ArImage_getPlaneData(mArSession, image, 2, &v, &vLength);
    
        ArImage_getPlaneRowStride(mArSession, image, 0, &yStride);
        ArImage_getPlaneRowStride(mArSession, image, 1, &uvStride);
        ArImage_getPlanePixelStride(mArSession, image, 1, &uvPixelStride);
    
        int width, height;
        ArImage_getWidth(mArSession, image, &width);
        ArImage_getHeight(mArSession, image, &height);
    
        auto *argb8888 = new uint32_t[width * height];
        ConvertYUV420ToARGB8888(y, u, v, argb8888, width, height, yStride, uvStride, uvPixelStride);
    
        std::ofstream stream("/data/user/0/{your app package name}/cache/img", std::ios::out | std::ios::binary);
        for(int i = 0; i < width * height; i++)
          stream.write((char *) &argb8888[i], sizeof(uint32_t));
    
        stream.close();
        LOGD("%i %i", width, height);
    
        delete[](argb8888);
      }
      if (image != nullptr) {
        ArImage_release(image);
      }
    

    However, I did one another thing to apply Tensorflow yuv2rgb code for my purpose. YUV2RGB inside yuv2rgb.cc have BRGA order, while Android ARGB_8888 have ARGB order. More shortly, in inline YUV2RGB method you need to change this line:

    return 0xff000000 | (nR << 16) | (nG << 8) | nB;
    

    to

    return 0xff000000 | nB << 16 | nG << 8 | nR;