opencvyuvdistortionremap

How to undistort I420 image data? Efficiently

I am able to undistort RGB image successfully.

Now, I am working on directly undistort I420 data, instead of first converting it to RGB.

Below are the steps I followed after camera calibration.

K = cv::Matx33d(541.2152931632737, 0.0, 661.7479652584254,
                0.0, 541.0606969363056, 317.4524205037745,
                0.0,               0.0,               1.0);
D = cv::Vec4d(-0.042166406281296365, -0.001223961942208027, -0.0017036710622692108, 0.00023929900459453295);
newSize = cv::Size(3400, 1940);
cv::Matx33d new_K;
cv::fisheye::estimateNewCameraMatrixForUndistortRectify(K, D, cv::Size(W, H), cv::Mat::eye(3, 3, CV_64F), new_K, 1, newSize);    // W,H are the distorted image size
cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat::eye(3, 3, CV_64F), new_K, newSize, CV_16SC2, mapx, mapy);

cv::remap(src, dst, mapx, mapy, cv::INTER_LINEAR);

Above code is giving me undistorted image successfully.

Now I want to undistort I420 data. So, now my src will be an I420/YV12 data. How can I undistort an I420 data, without converting it first to RGB?

By the way I420 is an image format with only 1 channel(unlike 3 channels in RGB). It has height = 1.5*image height. Its width is equal to image width.

Below code is to convert I420 to BGR

cvtColor(src, BGR, CV_YUV2BGR_I420, 3);

BGR - pixel arrangement BGR I420 - pixel arrangement I

Solution

  • The most efficient solution is resizing mapx and mapy and applying shrunk maps on down-sampled U and V channels:

    Here is a Python OpenCV sample code (please read the comments): 

    import cv2 as cv
import numpy as np

# For the example, read Y, U and V as separate images.
srcY = cv.imread('DistortedChessBoardY.png', cv.IMREAD_GRAYSCALE) #  Y color channel (1280x720)
srcU = cv.imread('DistortedChessBoardU.png', cv.IMREAD_GRAYSCALE) #  U color channel (640x360)
srcV = cv.imread('DistortedChessBoardV.png', cv.IMREAD_GRAYSCALE) #  V color channel (640x360)

H, W = srcY.shape[0], srcY.shape[1]

K = np.array([[541.2152931632737, 0.0, 661.7479652584254],      
              [0.0, 541.0606969363056, 317.4524205037745],
              [0.0,               0.0,               1.0]])

D = np.array([-0.042166406281296365, -0.001223961942208027, -0.0017036710622692108, 0.00023929900459453295])

# newSize = cv::Size(3400, 1940);
newSize = (850, 480)

# cv::Matx33d new_K;
new_K = np.eye(3)

# cv::fisheye::estimateNewCameraMatrixForUndistortRectify(K, D, cv::Size(W, H), cv::Mat::eye(3, 3, CV_64F), new_K, 1, newSize);    // W,H are the distorted image size
new_K = cv.fisheye.estimateNewCameraMatrixForUndistortRectify(K, D, (W, H), np.eye(3), new_K, 1, newSize)

# cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat::eye(3, 3, CV_64F), new_K, newSize, CV_16SC2, mapx, mapy);
mapx, mapy = cv.fisheye.initUndistortRectifyMap(K, D, np.eye(3), new_K, newSize, cv.CV_16SC2);

# cv::remap(src, dst, mapx, mapy, cv::INTER_LINEAR);
dstY = cv.remap(srcY, mapx, mapy, cv.INTER_LINEAR)

# Resize mapx and mapy by a factor of x2 in each axis, and divide each element in the map by 2
shrank_mapSize = (mapx.shape[1]//2, mapx.shape[0]//2)
shrunk_mapx = cv.resize(mapx, shrank_mapSize, interpolation = cv.INTER_LINEAR) // 2
shrunk_mapy = cv.resize(mapy, shrank_mapSize, interpolation = cv.INTER_LINEAR) // 2

# Remap U and V using shunk maps
dstU = cv.remap(srcU, shrunk_mapx, shrunk_mapy, cv.INTER_LINEAR, borderValue=128)
dstV = cv.remap(srcV, shrunk_mapx, shrunk_mapy, cv.INTER_LINEAR, borderValue=128)

cv.imshow('dstY', dstY)
cv.imshow('dstU', dstU)
cv.imshow('dstV', dstV)

cv.waitKey(0)
cv.destroyAllWindows()

    Result:

    Y:
    dstY

    U:
    dstU

    V:
    dstV

    After converting to RGB:
    RGB

    C++ implementation considerations:

    Since I420 format arranges Y, U and V as 3 continuous planes in memory, it's simple to set a pointer to each "plane", and treat it as a Grayscale image.
    Same data ordering applies the output image - set 3 pointer to output "planes".

    Illustration (assuming even width and height, and assume byte stride equals width): 

    srcY -> YYYYYYYY           dstY -> YYYYYYYYYYYY
        YYYYYYYY                   YYYYYYYYYYYY
        YYYYYYYY                   YYYYYYYYYYYY
        YYYYYYYY                   YYYYYYYYYYYY
        YYYYYYYY   remap           YYYYYYYYYYYY
        YYYYYYYY  ======>          YYYYYYYYYYYY
srcU -> UUUU                       YYYYYYYYYYYY
        UUUU               dstU -> YYYYYYYYYYYY
        UUUU                       UUUUUU
srcV -> VVVV                       UUUUUU
        VVVV                       UUUUUU
        VVVV                       UUUUUU
                           dstV -> VVVVVV
                                   VVVVVV
                                   VVVVVV
                                   VVVVVV

    Implementing above illustration is C++

    Under the assumption that width and height are even, and byte stride equals width, you can use the following C++ example for converting I420 to Y, U and V planes:

    Assume: srcI420 is Wx(H*3/2) matrix in I420 format, like cv::Mat srcI420(cv::Size(W, H * 3 / 2), CV_8UC1);. 

    int W = 1280, H = 720;  //Assume resolution of Y plane is 1280x720

//Pointer to Y plane
unsigned char *pY = (unsigned char*)srcI420.data;

//Y plane as cv::Mat, resolution of srcY is 1280x720
cv::Mat srcY = cv::Mat(cv::Size(W, H), CV_8UC1, (void*)pY);

//U plane as cv::Mat, resolution of srcU is 640x360 (in memory buffer, U plane is placed after Y).
cv::Mat srcU = cv::Mat(cv::Size(W/2, H/2), CV_8UC1, (void*)(pY + W*H));

//V plane as cv::Mat, resolution of srcV is 640x360 (in memory buffer, V plane is placed after U).
cv::Mat srcV = cv::Mat(cv::Size(W / 2, H / 2), CV_8UC1, (void*)(pY + W*H + (W/2*H/2)));

//Display srcY, srcU, srcV for testing
cv::imshow("srcY", srcY);
cv::imshow("srcU", srcU);
cv::imshow("srcV", srcV);
cv::waitKey(0);

    Above example uses pointer manipulations, without the need for copying the data.
    You can use the same pointer manipulations for your destination I420 image.

    Note: The solution is going to work in most cases, but not guaranteed to work in all cases.