[SOLVED] Image matching fails with low confidence using pyautogui and OpenCV

Image matching fails with low confidence using pyautogui and OpenCV

I'm working on automating GUI testing using OpenCV and PyAutoGui. I tried both pyautogui.locateOnScreen() and cv2.matchTemplate() to detect UI elements by matching a reference image inside a screen region.

The reference image is visibly present inside the screenshot (confirmed manually), but both approaches either fail or return very low confidence (~0.21), and no click is triggered.

The image path is passed via sys.argv.

if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("[ERROR] Usage: python OCR.py <word> <confidence>")
    else:
        word = sys.argv[1]
        reference_image = sys.argv[2]
        result = find_and_click_text(word, reference_image)
        print(result)

Here's what works:

If the path is hardcoded inside the .py script → locateOnScreen() works perfectly.
If I pass the image path via sys.argv, and print it → it looks identical.
Image.open(reference_image) works — file exists and loads fine.

Here's what fails:

pyautogui.locateOnScreen(reference_image) → fails silently or gives low confidence (~0.2)
Even if I pass the loaded image object (Image.open(path)) to locateOnScreen() → it fails
OpenCV's cv2.matchTemplate() also gives low confidence (~0.21) even though the reference image clearly appears in the screenshot.

.py File Output:

[DEBUG] Reference image path received: C:\PFE\Test_Automation\CubeMX\PA6_reference.png
[DEBUG] sys.argv: ['OCR.py', 'GPIOOutput', 'C:\\PFE\\Test_Automation\\CubeMX\\PA6_reference.png']
[DEBUG] File exists at: C:/PFE/Test_Automation/CubeMX/PA6_reference.png
[ERROR] Traceback (most recent call last):
  File "C:\Python312\Lib\site-packages\pyautogui\__init__.py", line 172, in wrapper
    return wrappedFunction(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\pyautogui\__init__.py", line 210, in locateOnScreen
    return pyscreeze.locateOnScreen(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\pyscreeze\__init__.py", line 405, in locateOnScreen
    retVal = locate(image, screenshotIm, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\pyscreeze\__init__.py", line 383, in locate
    points = tuple(locateAll(needleImage, haystackImage, **kwargs))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python312\Lib\site-packages\pyscreeze\__init__.py", line 257, in _locateAll_opencv
    raise ImageNotFoundException('Could not locate the image (highest confidence = %.3f)' % result.max())
pyscreeze.ImageNotFoundException: Could not locate the image (highest confidence = 0.274)

What I’ve Confirmed:

The problem isn't solved even when I limit the search area using pyautogui.screenshot(region=(...))
ImageChops.difference() confirms the reference and cropped screen area are pixel-perfect
The reference image isn't accessed after running the script.

Screenshots: The first image is my screen and the second is the reference image.

Thanks in advance — I’m happy to share any additional information (code + screenshots) if needed.

Solution

Here is your template overlaid side by side with the instance I think you want to locate.

See how the image content is not the same size? That is the problem. That can't match. It has to be the same size/scale.

Pick the template correctly. Then it will work.