I'm trying to reconstruct position of chessboard from its stereo correspondences taken from calibrated stereo camera images. According to this answer, I went through whole process as
I don't understand one step, that is WHY should I rectify stereo images during stereoCalibration using openCV's cv2::stereoRectify() or Matlab's rectifyStereoImages. Or maybe better: HOW should I get projection matrices - by stereo rectification or simply as
P = K×[R|t]
from R, t and K gotten from stereoCalib / cameraCalib function.
As I understood the topic, stereo rectification is somehow important for making depth map, but as I've only ever worked with 3D estimation by triangulation, I am not sure if it is anyhow useful in this case. I've got a bit confused because every single stereo calibration demo I found contains this rectification step.
More complex answers are welcome as I think I figured the answer is simply calculate it by provided equation, but I would like to understand more what this rectification is about.
If you already know which points correspond to each other on two images you are right, stereo rectification is not needed. You can just triangulate the points to get the 3D coordinate.
But if you are trying to estimate a depth map from a whole image you will need to solve the stereo correspondence problem, i.e. you will need to find for each point in one image the corresponding point in the second image.
By stereo rectifying you ensure that epipolar lines are horizontal and corresponding points only can be on the same horizontal pixel row in both images. This means your search space for correspondences is radically smaller and thus the problem becomes tractable.
Let me know if this answers your question or if you need a few more pointers into the right direction!