I already do rectification with my own code. Now I am trying to make cv2.stereoRectify work.
Suppose I have the following code:
import numpy as np
import cv2
img1 = cv2.imread(IMG_LEFT) # Load image left
img2 = cv2.imread(IMG_RIGHT) # Load image right
A1 = np.array(A1) # Left camera matrix intrinsic
A2 = np.array(A2) # Right camera matrix intrinsic
RT1 = np.array(RT1) # Left camera extrinsic (3x4)
RT2 = np.array(RT2) # Right camera extrinsic (3x4)
# Original projection matrices
Po1 = A1.dot( RT1 )
Po2 = A2.dot( RT2 )
# Camera centers (world coord.)
C1 = -np.linalg.inv(Po1[:,:3]).dot(Po1[:,3])
C2 = -np.linalg.inv(Po2[:,:3]).dot(Po2[:,3])
# Transformations
T1to2 = C2 - C1 # Translation from first to second camera
R1to2 = RT2[:,:3].dot(np.linalg.inv(RT1[:,:3])) # Rotation from first to second camera (3x3)
Then, I would like to find the rectification transformations (3x3). Following the OpenCV documentation I am trying:
Rectify1, Rectify2, Pn1, Pn2, _, _, _ = cv2.stereoRectify(A1, np.zeros((1,5)), A2, np.zeros((1,5)), (img1.shape[1], img1.shape[0]), R1to2, T1to2, alpha=-1 )
mapL1, mapL2 = cv2.initUndistortRectifyMap(A1, np.zeros((1,5)), Rectify1, Pn1, (img1.shape[1], img1.shape[0]), cv2.CV_32FC1)
mapR1, mapR2 = cv2.initUndistortRectifyMap(A2, np.zeros((1,5)), Rectify2, Pn2, (img2.shape[1], img2.shape[0]), cv2.CV_32FC1)
img1_rect = cv2.remap(img1, mapL1, mapL2, cv2.INTER_LINEAR)
img2_rect = cv2.remap(img2, mapR1, mapR2, cv2.INTER_LINEAR)
Anyway I am getting totally screwed images, surely not rectified. What am I doing wrong? I guess it is something about rotations/translations, but I cannot figure it out.
Also, is OpenCV a bit overcomplicated about this? It should be an easy operation anyway.
Many thanks.
EDIT: You may notice that I set distortion parameters to zero. That is because I am using computer generated stereo images that have no lens distortion.
Digging into OpenCV documentation I found the reason why stereoRectify() does not seem to work.
Most of research paper often refer to homography transformation to be applied to the image.
OpenCV, instead, is computing the rotation in the object space as it is (shyly) explained in the cv2.initUndistortrectifyMap() documentation (see here).
So after calling:
R1, R2, Pn1, Pn2, _, _, _ = cv2.stereoRectify(A1, np.zeros((1,5)), A2, np.zeros((1,5)), (img1.shape[1], img1.shape[0]), R1to2, T1to2, alpha=-1 )
To get the rectification transformations I use:
Rectify1 = R1.dot(np.linalg.inv(A1))
Rectify2 = R2.dot(np.linalg.inv(A2))
Where R1 and R2 are the transformations output of OpenCV and A1 and A2 are the 3x3 camera matrices (intrinsics).
It seems to work just fine. Please comment if you have any better idea.
Hope this to be useful for anyone.