OpenCV camera calibration is inconsistent

I am trying to calibrate my camera using OpenCV. I am using the following guide:

https://docs.opencv.org/4.8.0/d4/d94/tutorial_camera_calibration.html

However I am finding that the results I get for the camera matrix depend heavily on how far the calibration images are from the camera. Is this supposed to be the case? There doesn’t seem to be any discussion as to how far the images should be from the camera in the guide.

Ultimately I want to use the camera matrix that I get from calibration in a call to solvePnP to get the pose of an object. If the camera matrix depends on the position of the calibration grid then the results of the solvePnP calculation will also depend on that.

What am I missing?

Camera calibration is fundamentally a data fitting procedure, similar to least squares fitting of a line to data. A good fit requires enough quality data to constraint the model well. Otherwise, the smallest amount of noise affects the parameters. I suspect that you get very uncertain results when calibrating with a checkerboard far away, which therefor only covers a small part of the image. A few tips:

• View of the checkerboard should ideally cover all of or large parts of the images. Otherwise, the estimated parameters are confounded with the estimated board pose. ChArUco and other coded targets allow for detection of partly visible targets.
• The mathematical camera model should only be as flexible as necessary - to avoid over-fitting. Most normal lenses can be accurately modeled with just one radial distortion parameter in the Brown-Conrady model used in OpenCV.
• For the focal length and principle point estimation, foreshortening must be observed (tilted board views).
• For good lens distortion estimation, the entire sensor area should be covered.
• Use the extended overload of cv::calibrateCamera() which provides an estimate of parameter uncertainties in stdDeviationsIntrinsics. As a rule of thumb, these uncertainties should be small compared to the parameter value.
• A professional software like Calib.io's Calibrator allows you to get the uncertainty in image or 3D space, which ultimately is the interesting quality factor.
• There is more, in-depth information here (I am the author): https://calib.io/blogs/knowledge-base/calibration-best-practices and https://calib.io/blogs/knowledge-base/understanding-parameter-uncertainty