opencvkinect-sdkazurekinect

How to convert k4a_image_t to opencv matrix? (Azure Kinect Sensor SDK)

I started playing around with Azure Kinect Sensor SDK. I went through the official how-to guides. I can capture images as raw buffers from the sensor, but I would like to turn them into opencv matrices.

Solution

  • First, you need to tell Azure Kinect sensor to capture in BGRA32 format for the color image (instead of JPEG or other compressed formats). The depth image is captured in 16 bit 1 channel format.

    You do this by setting up the config:

    k4a_device_configuration_t config = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL;
config.camera_fps = K4A_FRAMES_PER_SECOND_30;
config.color_format = K4A_IMAGE_FORMAT_COLOR_BGRA32; // <==== For Color image
config.color_resolution = K4A_COLOR_RESOLUTION_2160P;
config.depth_mode = K4A_DEPTH_MODE_NFOV_UNBINNED; // <==== For Depth image

    Then, once configured, you can capture a color image the following way, and then using the raw buffer create an opencv matrix from the color image:

    k4a_image_t colorImage = k4a_capture_get_color_image(capture); // get image metadata
if (colorImage != NULL)
{
    // you can check the format with this function
    k4a_image_format_t format = k4a_image_get_format(colorImage); // K4A_IMAGE_FORMAT_COLOR_BGRA32 

    // get raw buffer
    uint8_t* buffer = k4a_image_get_buffer(colorImage);

    // convert the raw buffer to cv::Mat
    int rows = k4a_image_get_height_pixels(colorImage);
    int cols = k4a_image_get_width_pixels(colorImage);
    cv::Mat colorMat(rows , cols, CV_8UC4, (void*)buffer, cv::Mat::AUTO_STEP);

    // ...

    k4a_image_release(colorImage);
}

    Similarly, for the depth image, you can convert the raw depth data to opencv like this (note, the matrix type has changed!):

    k4a_image_t depthImage = k4a_capture_get_depth_image(capture); // get image metadata
if (depthImage != NULL)
{
    // you can check the format with this function
    k4a_image_format_t format = k4a_image_get_format(depthImage); // K4A_IMAGE_FORMAT_DEPTH16 

    // get raw buffer
    uint8_t* buffer = k4a_image_get_buffer(depthImage);

    // convert the raw buffer to cv::Mat
    int rows = k4a_image_get_height_pixels(depthImage);
    int cols = k4a_image_get_width_pixels(depthImage);
    cv::Mat depthMat(rows, cols, CV_16U, (void*)buffer, cv::Mat::AUTO_STEP);

    // ...

    k4a_image_release(depthImage);
}

    NOTE: Opencv matrix constructor will not copy or allocate new memory for the pointer, instead it initializes the matrix head to point to the specified data!

    Sources: