c++opencvcomputer-visiontemplate-matchinggame-automation

OpenCV Template Matching for Playing Cards

For my bridge training lessons I 'm trying to analyse a deal played in Bridge Base Online, by capturing the image and then try to find where are the 52 cards.

The reference image is this one (capture.jpg) enter image description here

And I have also 52 images (41x68) of the cards, like the five of spades:

enter image description here

Now when doing pattern matching in OpenCV:

Mat1f result;
matchTemplate(org_gray, template_gray, result, TM_CCOEFF_NORMED);
double thresh = 0.8;
threshold(result, result, thresh, 1., THRESH_BINARY);

Mat1b resb;
result.convertTo(resb, CV_8U, 255);

std::vector<std::vector<Point>> contours;
findContours(resb, contours, RETR_LIST, CHAIN_APPROX_SIMPLE);

for (int i = 0; i < contours.size(); ++i)
{
    Mat1b mask(result.rows, result.cols, uchar(0));
    drawContours(mask, contours, i, Scalar(255), cv::FILLED);

    Point max_point,min_point;
    double max_val;
    minMaxLoc(result, NULL, &max_val, &min_point, &max_point, mask);

    rectangle(img, Rect(max_point.x, max_point.y, ptpw->m.cols, ptpw->m.rows), Scalar(0, 255, 0), 2);
}
imshow("b", ptpw->m);
imshow("a",img);

The result is this one. It did detect the location of S5 but it also detected 7 more cards...

enter image description here

How can I enhance my algorithm?

If I raise the threshold to 0.87 it finds only one card, but the 5 of clubs instead. I can adjust the threshold to find only one card, but why the 5C instead of 5S?

Thanks.

Solution

  • As is the case with everyone who asks, the matching mode is bad. For perfect data without brightness variations, the TM_SQDIFF* modes are the best choice.

    matchTemplate(org_gray, template_gray, result, TM_SQDIFF_NORMED);
double thresh = 0.1;
threshold(result, result, thresh, 1., THRESH_BINARY_INV);

    You will get a difference if the instance is red/gray, so you should consider converting to grayscale by simply taking the green or blue channel of the image (red text on white background is just all bright in the red channel). You could do that by splitting the source channels or with mixChannels, or even with transform and a custom mixing matrix.

    Further, your template has a wide white border. It's too wide. On the expected perfect match, it overlaps onto the image of a neighboring card (black border, black number), reducing the quality of the match.

    Trim it down. That'll do better.

    enter image description here

    SQDIFF_NORMED on the original template:

    enter image description here

    With template cropped more closely:

    enter image description here

    Even here it's not perfect because of JPEG compression artefacts. I also think the card instance's rounded edge intrudes on the square template.

    And then you'll need Non-Maximum Suppression (NMS). If you just threshold the scores array, you'll get adjacent "on" pixels for the same detection, or even extrema that are almost adjacent for some reason. So don't just threshold, but do a little more. This is a general recipe with steps that were needed in various situations.

    Sketch in Python:

    nms_threshold = 0.10 # looking for minima
nms_radius = 5
localmin = cv.erode(scores, None, iterations=nms_radius)
extrema = (scores == localmin) & (scores <= nms_threshold)
extrema = cv.morphologyEx(extrema.astype(np.uint8), cv.MORPH_CLOSE, None, iterations=nms_radius)
# and then connected components, with stats for centroid
# will also handle minima larger than 1 pixel
(nlabels, labels, stats, centroids) = cv.connectedComponentsWithStats(extrema.astype(np.uint8))