How to get the OpenCV image from Python and use it in C++ in pybind11?

I'm trying to figure out how it is possible to receive an OpenCV image from a Python in C++. I'm trying to send a callback function, from C++ to my Python module, and then when I call a specific python method in my C++ app, I can access the needed image.

Before I add more details, I need to add that there are already several questions in this regard including :

1. how-to-convert-opencv-image-data-from-python-to-c
2. pass-image-data-from-python-to-cvmat-in-c
3. writing-python-bindings-for-c-code-that-use-opencv
4. c-conversion-from-numpy-array-to-mat-opencv

but none of them have anything about Pybind11. In fact they are all using the PyObject (from Python.h header) with and without Boost.Python. So my first attempt is to know how it is possible in Pybind11 knowing that it has support for Numpy arrays, so it can hopefully make things much easier.

Also On the C++ side, OpenCV has two versions, 3.x and 4.x which 4.x as I've recently found, is C++11 compliant. on Python side, I used OpenCV 3.x and I'm on a crossroad of which one to choose and what implications it has when it comes to Pybind11.

What I have tried so far: I made a quick dummy callback and tried passing a simple cv::Mat& like this :

#include <pybind11/embed.h>
#include <pybind11/numpy.h>
#include <pybind11/stl.h>
#include <pybind11/functional.h>
namespace py = pybind11;
...

void cpp_callback1(bool i, std::string id, cv::Mat img)
{ 
    auto timenow = chrono::system_clock::to_time_t(chrono::system_clock::now());
    cout  <<"arg1: " << i << " arg2: " << id<<" arg3: " << typeid(img).name() <<" " << ctime(&timenow)<<endl;
}

and used it like this :

py::list callback_lst;
callback_lst.attr("append")(py::cpp_function(cpp_callback1));

py::dict core_kwargs = py::dict("callback_list"_a = callback_lst,
                                "debug_show_feed"_a = true);

py::object core_obj = core_cls(**core_kwargs);
core_obj.attr("start")();

but it fails with an exception on python part which says :

29/03/2020 21:56:47 : exception occured ("(): incompatible function arguments. The following argument types are supported:\n    1. (arg0: bool, arg1: str, arg2: cv::Mat) -> None\n\nInvoked with: True, '5', array([[[195, 217, 237],\n        [195, 217, 237],\n        [196, 218, 238],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       [[195, 217, 237],\n        [195, 217, 237],\n        [195, 217, 237],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       [[195, 217, 237],\n        [195, 217, 237],\n        [195, 217, 237],\n        ...,\n        [211, 241, 255],\n        [211, 241, 255],\n        [211, 241, 255]],\n\n       ...,\n\n       [[120, 129, 140],\n        [110, 120, 130],\n        [113, 122, 133],\n        ...,\n        [196, 209, 245],\n        [195, 207, 244],\n        [195, 207, 244]],\n\n       [[120, 133, 142],\n        [109, 121, 130],\n        [114, 120, 131],\n        ...,\n        [195, 208, 242],\n        [195, 208, 242],\n        [195, 208, 242]],\n\n       [[121, 134, 143],\n        [106, 119, 128],\n        [109, 114, 126],\n        ...,\n        [194, 207, 241],\n        [195, 208, 242],\n        [195, 208, 242]]], dtype=uint8)",) 
Traceback (most recent call last):
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 257, in start
    self._main_loop()
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 301, in _main_loop
    self._execute_callbacks(is_valid, name, frame)
  File "C:\Users\Master\Anaconda3\Lib\site-packages\F\utils.py", line 142, in _execute_callbacks
    callback(*args)
TypeError: (): incompatible function arguments. The following argument types are supported:
    1. (arg0: bool, arg1: str, arg2: cv::Mat) -> None

Invoked with: True, '5', array([[[195, 217, 237],
        [195, 217, 237],
        [196, 218, 238],
        ...,
        [211, 241, 255],
        [211, 241, 255],
        [211, 241, 255]],

       [[195, 217, 237],
        [195, 217, 237],
        [195, 217, 237],
        ...,

Using py::object or py::array_t<uint8_t> instead of cv::Mat doesn't cause any errors, but I can't seem to find a way to cast them back to a cv::Mat properly!

I tried to cast the numpy array into a cv::Mat as instructed in the comments but the output is garbage:

void cpp_callback1(bool i, std::string id, py::array_t<uint8_t>& img)
{ 
    auto im = img.unchecked<3>();
    auto rows = img.shape(0);
    auto cols = img.shape(1);
    auto type = CV_8UC3;

    //py::buffer_info buf = img.request();
    cv::Mat img2(rows, cols, type, img.ptr());
    cv::imshow("test", img2);
}

results in :

It seems to me, the strides, or something in that direction is messed up that image is showing like this. what am I doing wrong here? I couldn't use the img.strides() though! when printed it using py::print, it shows 960 or something like that. So I'm completely clueless how to interpret that!

I ultimately could successfully get this to work thanks to @DanMasek and this link:

void cpp_callback1(py::array_t<uint8_t>& img)
{ 
    py::buffer_info buf = img.request();
    cv::Mat mat(buf.shape[0], buf.shape[1], CV_8UC3, (unsigned char*)buf.ptr);
    
    cv::imshow("test", mat);
}

note that the cast is necessary, or otherwise, you'd get a blackish screen only!
However, if somehow there was a way like py::return_value_policy that we could use to change the type of reference, so even though the python part ends, the c++ side wouldn't crash would be great.

side note :
it seems the ptr property exposed in the numpy array, is actually not a py::handle but a PyObject*&. I couldn't have a successful conversion and thus resorted to the solution I posted above. I'll update this answer, when I figure this out.

Update:

I found out, the arrays data holds a pointer to the underlying buffer and can be used easily as well. From <pybind11/numpy.h> L681:

/// Pointer to the contained data. If index is not provided, points to the
/// beginning of the buffer. May throw if the index would lead to out of bounds access.

So my original code that used img.ptr(), can work using img.data() like this :

void cpp_callback1(py::array_t<uint8_t>& img)
{ 
    //auto im = img.unchecked<3>();
    auto rows = img.shape(0);
    auto cols = img.shape(1);
    auto type = CV_8UC3;

    cv::Mat img2(rows, cols, type, (unsigned char*)img.data());
    cv::imshow("test", img2);
}