matlabstereo-3dcalibration3d-reconstructionstereoscopy

Stereo Calibration and 3D Reconstruction Issue

I am trying to perform stereo camera calibration and 3D reprojection of a calculated disparity image.
For the sake of simplicity I made this example in Matlab but getting the same result in Python and OpenCV.

I use this piece of code for the calibration:

% Detect checkerboards in images
[imagePoints, boardSize, imagesUsed] = detectCheckerboardPoints(imageFileNames_left, imageFileNames_right);

% Generate world coordinates of the checkerboard keypoints
squareSize = 108;  % in units of 'millimeters'
worldPoints = generateCheckerboardPoints(boardSize, squareSize);

% Read one of the images from the first stereo pair
I1 = imread(imageFileNames_left{1});
[mrows, ncols, ~] = size(I1);

% Calibrate the camera
[stereoParams, pairsUsed, estimationErrors] = estimateCameraParameters(imagePoints, worldPoints, ...
    'EstimateSkew', false, 'EstimateTangentialDistortion', true, ...
    'NumRadialDistortionCoefficients', 3, 'WorldUnits', 'millimeters', ...
    'InitialIntrinsicMatrix', [], 'InitialRadialDistortion', [], ...
    'ImageSize', [mrows, ncols]);

The image_FileNames contains the paths to the respective calibration images. I use a checkerboard calibration pattern printed on a rigid panel of size A0.

The resluts of the calibration look very promising:

However calculating the disparity and reprojecting it to 3D gives weird results:

Link to reprojected point cloud image

I used the following code snippet for the projection to 3D:

I1 = imread(strcat(dirs_left{i}, '/', names_left{i}));
I2 = imread(strcat(dirs_right{i}, '/', names_right{i}));

% Rectify using calibration data
[J1, J2] = rectifyStereoImages(I1,I2,stereoParams);

% Calculate Disparity
disparityRange = [0 128];
disparityMap = disparitySGM(rgb2gray(J1),rgb2gray(J2),'DisparityRange', disparityRange);

% Project to 3D
point3D = reconstructScene(disparityMap, stereoParams);

% Convert from millimeters to meters.
point3D = point3D ./ 1000;

% Visualize the 3-D Scene
ptCloud = pointCloud(point3D, 'Color', J1);
h1 = figure; pcshow(ptCloud);

The corresponding disparity image (Calculated using SGBM) looks great:

Link to disparity image

And here is the calibration data: (Attention: OpenCV notation, I transposed all matrices to be compatible with OpenCV):

camera matrix 0:
 - 2362.9276056, 0.0000006, 1034.4700766,
 - 0.0000006, 2366.4078916, 728.4543626,
 - 0.0000006, 0.0000006, 1.0000006,

camera matrix 1:
 - 2366.2683866, 0.0000006, 1030.6057166,
 - 0.0000006, 2366.4804296, 740.0748076,
 - 0.0000006, 0.0000006, 1.0000006,

lens dist 0: ()
-0.201011, 0.094025, -0.000569, 0.000521, 0.252866

lens dist 1:
-0.191647, 0.046607, -0.000569, 0.000521, 0.205665

rotation matrix camera 1:
 - 0.9994606, -0.0068816, 0.0321416,
 - 0.0055786, 0.9991666, 0.0404456,
 - -0.0323936, -0.0402446, 0.9986656,

translation vector camera 1:
 - -485.0037626, -48.0975216, 48.2236646,

I get a similar point cloud using OpenCV instead of Matlab for the re-projection, the above calibration info.

Also I found this question, where they seem to get a similar output after the projection. But in my case all the reprojection errors are rather small (0.26pixels in average) and the disparity range is set by default to 0-128, which is the maximum, and the disparity map looks great.

Any ideas?

Solution

  • The predicted camera paramters match the physical properties of the camera lens used. This led me to limiting the projection space, basically as fdermishin suggested. Removing all points outside a 45 m distance gave reasonable results!

    I used the following piece of code to filter the points:

    roi = [-50 50 -10 40 0 45]; % filter x, y, and z
indices = findPointsInROI(ptCloud,roi);
ptCloud_clean = select(ptCloud,indices);