I am trying to compute surface normals in OpenCV. Well, this should be quick and easy but I don't know why it is not working. Below is the code:
> import cv2
> img_color = cv2.imread("color.png")
> img_depth = cv2.imread("depth.png", cv2.CV_16UC1) # millimeters
> img_color.shape, img_color.dtype
((720, 1280, 3), dtype('uint8'))
> img_depth.shape, img_depth.dtype
((720, 1280), dtype('uint16'))
> k = np.array([[ fx, 0, cx ],
[ 0, fy, cy ],
[ 0, 0, 1 ]])
> k
array([[900, 0, 640],
[ 0, 900, 360],
[ 0, 0, 1]])
> img_depth_m = img_depth.astype('float32') * 0.001 # meter
> rows, cols = img_depth_m.shape
> normal_estimator = cv2.rgbd.RgbdNormals_create(rows, cols, cv2.CV_32F, k, 3)
> normals = normal_estimator.apply( img_depth_m )
It throws the following error:
error Traceback (most recent call last)
/tmp/ipykernel_19178/1208043521.py in <module>
----> 4 normals = normal_estimator.apply( img_depth_m )
error: OpenCV(4.2.0) ../contrib/modules/rgbd/src/normal.cpp:776:
error: (-215:Assertion failed) points3d_ori.channels() == 3 in function 'operator()'
It seems that OpenCV is expecting a 3 Channel input matrix. However, I looked at the docstring in the notebook, it says:
Docstring:
apply(points[, normals]) -> normals
. Given a set of 3d points in a depth image, compute the normals at each point.
. * @param points a rows x cols x 3 matrix of CV_32F/CV64F or a rows x cols x 1 CV_U16S
. * @param normals a rows x cols x 3 matrix
Type: builtin_function_or_method
Anyway, how to compute surface normals using camera intrinsic matrix in OpenCV?
PS: I am using OpenCV v4.2.0 on Ubuntu 20.04 LTS.
There is a function called depthTo3d in OpenCV to do convert depth image into 3D points. Please see the following code snippet:
In [1]: import cv2
In [2]: cv2.rgbd.depthTo3d?
Docstring:
depthTo3d(depth, K[, points3d[, mask]]) -> points3d
. Converts a depth image to an organized set of 3d points.
. * The coordinate system is x pointing left, y down and z away from the camera
. * @param depth the depth image (if given as short int CV_U, it is assumed to be the depth in millimeters
. * (as done with the Microsoft Kinect), otherwise, if given as CV_32F or CV_64F, it is assumed in meters)
. * @param K The calibration matrix
. * @param points3d the resulting 3d points. They are of depth the same as `depth` if it is CV_32F or CV_64F, and the
. * depth of `K` if `depth` is of depth CV_U
. * @param mask the mask of the points to consider (can be empty)
Type: builtin_function_or_method
In [3]: points3d = cv2.rgbd.depthTo3d(img_depth, k)
In [4]: normal_estimator = cv2.rgbd.RgbdNormals_create(rows, cols, cv2.CV_32F, k, 3)
In [5]: normals = normal_estimator.apply( points3d )