pythonopencvcomputer-visionaruco

How do to camera calibration using charuco board for OpenCV 4.10.0?


I am trying to do a camera calibration using OpenCV, version 4.10.0. I already got a working version for the usual checkerboard, but I can't figure out how it works with charuco. I would be grateful for any working code example.

What I tried: I tried following this tutorial: https://medium.com/@ed.twomey1/using-charuco-boards-in-opencv-237d8bc9e40d

It seems that essential functions like: cv.aruco.interpolateCornersCharuco and cv.aruco.interpolateCornersCharuco are missing. Even threw the documentation states an existing Python implementation, see: https://docs.opencv.org/4.10.0/d9/d6a/group__aruco.html#gadcc5dc30c9ad33dcf839e84e8638dcd1

I also tried following the official documentation for C++, see https://docs.opencv.org/4.10.0/da/d13/tutorial_aruco_calibration.html

The ArucoDetector in Python doesnt have the detectBoard method. So its also impossible to follow this tutorial in total.

But I guess from a hint in the documentation that the functions used by Medium are deprectated? But in no place marked as “removed”!

I already got the markers detected:

Detected image.

But then getting the object and image point fails:

`object_points_t, image_points_t = charuco_board.matchImagePoints( marker_corners, marker_ids)`

Any help or working code would be highly appreciated.

P.S.: My output of the “detectMarkers” method seems valid. Detected corners are of type

std::vector<std::vector<Point2f>.

(So translated to python an Array of Arrays containing the 4 points made by 2 coordinates each.) ID’s are

std::vector<int>

so in Python a list of Integers.

So I guess the python function “matchImagePoints” gets what it wants!

The marker detection seems succesful. I already tried changing the corner array: The detectMarkers method returns a tuple. I used the folloqing code to create the desired array of shape (X, 4, 2). (X beeing the number of deteced markers. Each has 4 corners with 2 coordinates x and y.)

marker_corners = np.array(marker_corners)
marker_corners = np.squeeze(marker_corners)

So I have the following:

marker_corners = [
[[8812. 5445.]
[8830. 5923.]
[8344. 5932.]
[8324. 5452.]],

[[7172. 5469.]
[7184. 5947.]
[6695. 5949.]
[6687. 5476.]],

[[3896. 5481.]
[3885. 5952.]
[3396. 5951.]
[3406. 5483.]],
...
]

marker_ids = [
[11],
[27],
[19],
...
]

Both, Passing the original return I get from

detector.detectMarkers

into the function and passing my modified array are failing. (Also not using squeeze and inputting a (X, 1, 4, 2) array fails!)

I can't get further anymore.

Minimum working example:

Use this picture: Charuco board 11x8

import cv2 as cv
import numpy as np
    
image = cv.imread("charuco_board.png")
im_gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY)
charuco_marker_dictionary = cv.aruco.getPredefinedDictionary(cv.aruco.DICT_6X6_250)
charuco_board = cv.aruco.CharucoBoard(
size=(11, 8),
squareLength=500,
markerLength=300,
dictionary=charuco_marker_dictionary
)

# Initial method of this question:
params = cv.aruco.DetectorParameters()
detector = cv.aruco.ArucoDetector(charuco_marker_dictionary, params)
marker_corners, marker_ids, rejected_candidates = detector.detectMarkers(im_gray)
marker_corners = np.array(marker_corners)
marker_corners = np.squeeze(marker_corners)

    # Using cv.aruco.CharucoDetector as pointed out in the comments.
    detector_charuco = cv.aruco.CharucoDetector(charuco_board)
    result = detector_charuco.detectBoard(im_gray)
    marker_corners_charuco, marker_ids_charuco = result[2:]

    # Compare the two results
    assert (marker_ids == marker_ids_charuco).all()  # Detected ID's are identical.
    # assert (marker_corners == marker_corners_charuco).all()  # There seems to be a difference.
    print(marker_corners[0:2], marker_corners_charuco[0:2])  # They seem to be in a different order.

# Proof of the different order statement:
def reshape_and_sort(array):
    array_reshaped = array.copy().reshape(-1, 2)
    return np.array(sorted(array_reshaped, key=lambda x: x[0]**2 + x[1]**2))  # Using geometric distance between each point and the point (0, 0). Leaving out the square.

marker_corners_reshaped = reshape_and_sort(marker_corners)
marker_corners_reshaped_charuco = reshape_and_sort(np.array(marker_corners_charuco))
assert (marker_corners_reshaped == marker_corners_reshaped_charuco).all()


# Trying with new resutls:  # Still fails!
try:
    object_points_t, image_points_t = charuco_board.matchImagePoints(
        marker_corners_charuco,
        marker_ids_charuco
    )
except cv.error as err:
    print(err)

Solution

  • I managed to get the calibration done with the official non-contrib opencv code. Here is a minimal working example:

    The problem: Be careful when defining your detection board. A 11x8 charuco board has a different order of the aruco markers as an 8x11. Even if they look very similar when printed. The detecion will fail.

    from typing import NamedTuple
    import math
    
    import matplotlib.pyplot as plt
    import cv2 as cv
    import numpy as np
    
    class BoardDetectionResults(NamedTuple):
        charuco_corners: np.ndarray
        charuco_ids: np.ndarray
        aruco_corners: np.ndarray
        aruco_ids: np.ndarray
    
    
    class PointReferences(NamedTuple):
        object_points: np.ndarray
        image_points: np.ndarray
    
    
    class CameraCalibrationResults(NamedTuple):
        repError: float
        camMatrix: np.ndarray
        distcoeff: np.ndarray
        rvecs: np.ndarray
        tvecs: np.ndarray
    
    
    SQUARE_LENGTH = 500
    MARKER_LENGHT = 300
    NUMBER_OF_SQUARES_VERTICALLY = 11
    NUMBER_OF_SQUARES_HORIZONTALLY = 8
    
    charuco_marker_dictionary = cv.aruco.getPredefinedDictionary(cv.aruco.DICT_6X6_250)
    charuco_board = cv.aruco.CharucoBoard(
    size=(NUMBER_OF_SQUARES_HORIZONTALLY, NUMBER_OF_SQUARES_VERTICALLY),
    squareLength=SQUARE_LENGTH,
    markerLength=MARKER_LENGHT,
    dictionary=charuco_marker_dictionary
    )
    
    image_name = f'ChArUco_Marker_{NUMBER_OF_SQUARES_HORIZONTALLY}x{NUMBER_OF_SQUARES_VERTICALLY}.png'
    charuco_board_image = charuco_board.generateImage(
            [i*SQUARE_LENGTH
             for i in (NUMBER_OF_SQUARES_HORIZONTALLY, NUMBER_OF_SQUARES_VERTICALLY)]
    )
    cv.imwrite(image_name, charuco_board_image)
    
    
    def plot_results(image_of_board, original_board, detection_results, point_references):
        fig, axes = plt.subplots(2, 2)
        axes = axes.flatten()
        img_rgb = cv.cvtColor(img_bgr, cv.COLOR_BGR2RGB)
        axes[0].imshow(img_rgb)
        axes[0].axis("off")
    
        axes[1].imshow(img_rgb)
        axes[1].axis("off")
        axes[1].scatter(
            np.array(detection_results.aruco_corners).squeeze().reshape(-1, 2)[:, 0],
            np.array(detection_results.aruco_corners).squeeze().reshape(-1, 2)[:, 1],
            s=5,
            c="green",
            marker="x",
        )
        axes[2].imshow(img_rgb)
        axes[2].axis("off")
    
        axes[2].scatter(
            detection_results.charuco_corners.squeeze()[:, 0],
            detection_results.charuco_corners.squeeze()[:, 1],
            s=20,
            edgecolors="red",
            marker="o",
            facecolors="none"
        )
        axes[3].imshow(cv.cvtColor(charuco_board_image, cv.COLOR_BGR2RGB))
        axes[3].scatter(
            point_references.object_points.squeeze()[:, 0],
            point_references.object_points.squeeze()[:, 1]
        )
        fig.tight_layout()
        fig.savefig("test.png", dpi=900)
        plt.show()
    
    
    def generate_test_images(image):
        """Use random homograpy.
    
        -> Just to test detection. This doesn't simulate a perspective
        projection of one single camera! (Intrinsics change randomly.)
        For a "camera simulation" one would need to define fixed intrinsics
        and random extrinsics. Then cobine them into a projective matrix.
        And apply this to the Image. -> Also you need to add a random z
        coordinate to the image, since a projection is from 3d space into 2d
        space.
        """
        h, w = image.shape[:2]
        src_points = np.float32([[0, 0], [w, 0], [w, h], [0, h]])
        dst_points = np.float32([
            [np.random.uniform(w * -0.2, w * 0.2), np.random.uniform(0, h * 0.2)],
            [np.random.uniform(w * 0.8, w*1.2), np.random.uniform(0, h * 0.6)],
            [np.random.uniform(w * 0.8, w), np.random.uniform(h * 0.8, h)],
            [np.random.uniform(0, w * 0.2), np.random.uniform(h * 0.8, h*1.5)]
        ])
        homography_matrix, _ = cv.findHomography(src_points, dst_points)
        image_projected = cv.warpPerspective(image, homography_matrix, (w, h))
        return image_projected
    
    
    def display_images(images):
        N = len(images)
        cols = math.ceil(math.sqrt(N))
        rows = math.ceil(N / cols)
    
        for i, img in enumerate(images):
            plt.subplot(rows, cols, i + 1)
            plt.imshow(img, cmap='gray')
            plt.axis('off')
        plt.tight_layout()
        plt.show()
    
    
    # Create N test images based on the originaly created pattern.
    N = 10
    random_images = []
    charuco_board_image = cv.cvtColor(charuco_board_image, cv.COLOR_GRAY2BGR)
    for _ in range(N):
        random_images.append(generate_test_images(charuco_board_image))
    display_images(random_images)
    
    
    total_object_points = []
    total_image_points = []
    for img_bgr in random_images:
        img_gray = cv.cvtColor(img_bgr, cv.COLOR_BGR2GRAY)
      charuco_detector = cv.aruco.CharucoDetector(charuco_board)
      detection_results = BoardDetectionResults(
          *charuco_detector.detectBoard(img_gray)
      )
    
        point_references = PointReferences(
            *charuco_board.matchImagePoints(
                detection_results.charuco_corners,
                detection_results.charuco_ids
            )
        )
        plot_results(
            img_gray,
            charuco_board_image,
            detection_results,
            point_references
        )
        total_object_points.append(point_references.object_points)
        total_image_points.append(point_references.image_points)
    
    
    calibration_results = CameraCalibrationResults(
        *cv.calibrateCamera(
            total_object_points,
            total_image_points,
            img_gray.shape,
            None,
            None
        )
    )
    
    
    
    """P.S.: Markers are too small in bigger pictures. They seem to not be adjustable.
    img_bgr_aruco = cv.aruco.drawDetectedMarkers(
        img_bgr.copy(),
        detection_results.aruco_corners
    )
    img_bgr_charuco = cv.aruco.drawDetectedCornersCharuco(
        img_bgr.copy(),
        detection_results.charuco_corners
    )
    """
    

    The other possibility is to install pip install opencv-contrib-python and not pip install opencv-python. At best make sure you install it in a new enviroment without any old installations.

    Here the following two essential functions will be available:

    cv.aruco.interpolateCornersCharuco
    cv.aruco.calibrateCameraCharuco
    

    For a more indepth explanation see here: