opencvopenniorbbec

How to use Orbbec Astra depth sensor with OpenCV?


I am currently working with an Orbbec Astra Mini depth sensor. I downloaded and installed the Astra Driver and OpenNI2 package. The depth sensor is shown in the device manager on my Windows 10 as soon as I attach it to the USB port. So far everything is fine.

My goal is to read RGB images and the depth map with OpenCV and show them with imshow.

The following code compiles fine in Visual Studio 2015, but I get this error:

OpenCV Error: Assertion failed (size.width>0 && size.height>0) in cv::imshow, file C:\build\master_winpack-build-win64-vc14\opencv\modules\highgui\src\window.cpp

... so I guess that OpenCV couldn't open the device and therefore my cv::Mat keeps being empty.

I somewhere read that I need to compile OpenCV with the OpenNI flag - is that true? If so, any useful links that might help me to make some progress? Any ideas what I might have missed?

#include<iostream>
#include<algorithm>
#include<fstream>
#include<System.h>
#include<time.h>
#include<opencv2/core/core.hpp>

using namespace std;

int main(int argc, char **argv) {

    cv::VideoCapture cap;
    cap.open(CV_CAP_OPENNI);
    cap.set(CV_CAP_OPENNI_IMAGE_GENERATOR_OUTPUT_MODE, CV_CAP_OPENNI_VGA_30HZ);
    cap.set(CV_CAP_OPENNI_DEPTH_GENERATOR_REGISTRATION, 1);

    cv::Mat im, dm;     // rgb image, depth map

    while (true)
    {

        if (cap.grab()) {
            cap.retrieve(im, CV_CAP_OPENNI_BGR_IMAGE);
            cap.retrieve(dm, CV_CAP_OPENNI_DEPTH_MAP);
        } else {
            cout << "ERROR: Could not grab image data." << endl;
        }

        if (!im.data) {
            cout << "ERROR: RGB not retrieved." << endl;
        }

        if (!dm.data) {
            cout << "ERROR: Depth map not retrieved." << endl;
        }

        cv::imshow("Image", im);
        cv::imshow("Depth", dm);

        if (cv::waitKey(1) >= 0)
            break;
    }

    cap.release();

    return 0;
}

UPDATE 19.02.18:

Issue solved by compiling the OpenCV libraries with the OpenNI flag as described here. Thank you Dmitrii!

However, the depth image is still quite weak. Any ideas where/how to adjust the parameters? Although, when running the sample applications of the Orbbec Astra SDK, the depth image is quite solid (which are working with OpenGL). So I guess the problem is somewhere within OpenCV?

Here's a screenshot of the depth map (inverted to white, quite weak resonance):

enter image description here


Solution

  • I wrote an comprehensive guide how to to compile ORB SLAM 2 (which was my preliminary goal) with OpenCV including OpenNI2 in order to use my Orbbec Astra camera as depth sensor. Hopefully someone else stumbling over this thread can use it.