OpenCV Stitching RGB and Depth (Intel RealSense)

I have a C++ application utilizing both realsense2 and opencv2 libraries. I'm pulling depth and RGB frames from two D415 camera. I'm able to stitch the RGB frames without issue; however, after much research I'm unable to find any example of how to duplicate steps from the RGB frame stitch to the depth frames. Does anyone know how to replicate stitching steps from one stitching job (RGB in my case) to another stitching job (depth in my case)?

I've looked at the Stitcher class reference documentation here: https://docs.opencv.org/master/d2/d8d/classcv_1_1Stitcher.html

If anyone has done this before and could provide some guidance that would be great.

Here is the working code I'm using to stitch (shortened):

// Start a streaming pipeline for each connected camera

for (auto && device : context.query_devices())

{

    string serial = device.get_info(RS2_CAMERA_INFO_SERIAL_NUMBER);

    pipeline pipeline(context);

    config config;

    config.enable_device(serial);

    config.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_BGR8, 15);

    config.enable_stream(RS2_STREAM_DEPTH, 1280, 720);

    config.enable_stream(RS2_STREAM_INFRARED, 1);

    pipeline.start(config);

    pipelines.emplace_back(pipeline);

    colorizers[serial] = colorizer();

}



map<int, frame> render_frames;



while(true)

{

    vector<Mat> mats;

    vector<frame> new_frames;

    for (auto && pipeline : pipelines)

    {

        frameset frameset = pipeline.wait_for_frames();

        if (frameset)

        {

            mats.push_back(frame_to_mat(frameset.get_color_frame()));

        }

    }

    Mat stitchedImage;

    Ptr<Stitcher> stitcher = Stitcher::create(Stitcher::PANORAMA);

    Stitcher::Status stitcherStatus = stitcher->stitch(mats, stitchedImage);

    if (stitcherStatus == Stitcher::OK)

    {

        imshow("Stitched Image", stitchedImage);
        auto k = waitKey(200);

    } else {

        cout << "Error stitching image.\n";

    }

}

Thanks!

If you ignore distortion then stitching is just computing a transform from one image origin to another. And technically you have everything you need already. Here's a VERY NAIVE approach.

Within each single camera (provided in the hardware specification) there is:

depth_origin -> rgb_origin
("->" represents a transform)

The OpenCV stitcher can estimate a transform between the two rgb images so you have

cam1_rgb_origin -> cam2_rgb_origin

So you can compute:

cam1_depth -> cam1_rgb -> cam2_rgb -> cam2_depth   

Your approach is a bit unusual but it might work sufficiently since the camera specs are provided in the realsense docs and I think each sensor is well aligned.

A bit of a warning though:

In general two cameras are never perfectly aligned. Hence for a more elaborate solution I'd avoid using the transform computed via image stitching. I'd rather calibrate the cameras on the static mount to obtain the extrinsic camera parameters. That is just the 3D transform from camera coordinate origin 1 to camera coordinate origin 2. Using this transform you'd create a 3D point cloud of the sensor data in a unified coordinate system containting both camera's points. Finally you can render this point cloud to a depth image from ANY perspective you choose and project the rgb images onto it to make it viable for the face detection you want to use. Camera calibration can be a royal pain especially when the camera's perspectives don't intersect a lot. So only follow this elaborate approach if your results don't suffice.

Sorry but for time sake I can't go into more detail. But I hope this points you in the right direction.