I am trying to configure a camera in pyBullet based on intrinsic and extrinsic parameters obtained from calibrating a real camera.
The camera is calibrated with OpenCV, giving me a camera matrix
|f_x 0 c_x|
| 0 f_y c_y|
| 0 0 1 |
and a vector of distortion coefficients
(k_1, k_2, p_1, p_2, k_3)
(I further have the pose of the camera but this is not relevant for the actual question, so I leave it out here.)
Unfortunately the computeProjectionMatrix function of pyBullet is a bit limited. It assumes f_x = f_y and c_x, c_y being exactly at the center of the image, which is both not true for my camera. Therefore I compute the projection matrix myself as follows (based on this):
projection_matrix = [
[2/w * f_x, 0, (w - 2c_x)/w, 0],
[0, 2/h * f_y, (2c_y - h)/h, 0],
[0, 0, A, B],
[0, 0, -1, 0],
]
where w,h are width and height of the image, A = (near + far)/(near - far) and B = 2 * near * far / (near - far), near and far defining the range on the z-axis that is included in the image (see pybullet.computeProjectionMatrix).
The above already gives me better results but the rendered images it still don't match exactly with the real images. I suspect one reason for this might be that the distortion is not taken into account.
So finally coming to my question:
How can I implement distortion for the simulated camera using the parameters I got from calibrating the real camera?
Is there a way I can integrate this in the projection matrix? If not, is there an other way?
As pointed out in the comments, non-linear distortion cannot be integrated into the matrices. What I am doing now is to first render the image without distortion and then distort the resulting image in a second step, using the code from this answer.
The image shrinks a bit due to the distortion, so when keeping the image size fixed there will be some empty area at the edges of the image. To compensate for this, I render the image at a slightly larger size than needed and then crop after distorting. Note that the centre point (c_x, c_y) needs to be adjusted accordingly, when increasing the size.
To illustrate with some pseudo code:
desired_image_size = (width, height)
# add 10% padding on each size
padding = desired_image_size * 0.1
render_image_size = desired_image_size + 2 * padding
# shift the centre point accordingly (other camera parameters are not changed)
c_x += padding[0]
c_y += padding[1]
# render image using the projection_matrix as described in the question
image = render_without_distortion(projection_matrix, camera_pose)
image = distort_image(image)
# remove the padding
image = image[padding[0]:-padding[0], padding[1]:-padding[1]]
This results in images that match very well with the ones from the real camera.
The complete implementation can be found here.