pythonpyautogui

pyautogui having trouble finding the given image

I'm using pyautogui.locateCenterOnScreen('image.png') to find/click a pink 'quest' button in Cornerpond.

Cornerpond

Even at ~0.85 confidence, it's clicking the same-sized, orange 'shop' button (no matches at 0.9).

import pyautogui
import win32gui

def locateImage():
    app = win32gui.FindWindow(None, 'Cornerpond')
    # region set this way because: 1920x1080 resolution, Cornerpond is 388x322 in the bottom right of my screen, right above the task bar.
    imageLocation = pyautogui.locateCenterOnScreen(r'C:\path\image.png', region=(1532,718,388,322), confidence=0.85)
    if imageLocation:
        print(imageLocation)
        pyautogui.click(imageLocation)
        
locateImage()

I tried finding all location results by using locateAllOnScreen to have a better idea of what's being flagged. It came up with 11 coordinates. 5 were the orange shop [$] button, 5 were the pink quest [!] button (the target image), and 1 was the green equipment button. The coordinates had a variance of x±1 or y±1, but all were 34x34.

import pyautogui

button_locations = list(pyautogui.locateAllOnScreen(r'C:\path\image.png', confidence = 0.85))

for location in button_locations:
    print(location)

At 0.85 confidence, I'm getting the correct location depending on the background/biome of Cornerpond. One biome gives the orange shop button, while another gives the target/pink quests button. The buttons do not move or change; only the background. What may be causing this, and what are my alternatives?

Solution

  • As I remeber in original question on Staging Ground there was button image with black border and background in corners.

    On Linux code works correctly when I crop button and remove border and background.
    This way button is more unique.

    button-min.png instead of button.png

    And it works even with confidence=0.99

    Code used for tests.

    Because I don't use Windows so I use locateAll() to test it on image from question.

    I used pillow to draw green/lime rectangles to check corrdinates from locateAll()

    import pyautogui
from PIL import Image, ImageDraw

# ---

print("pyautogui:", pyautogui.__version__)
print("OpenCV:", pyautogui.pyscreeze._useOpenCV)

# ---

screen = "screen.png"

# button = "button.png"  # with border  (shows two rectangles)
button = "button-min.png"  # without border  (shows one rectangle)
# button = "fish.png"
# button = "fish-min.png"    # without border
# button = "dollar.png"
# button = "dollar-min.png"  # without border

all_locations = pyautogui.locateAll(button, screen, confidence=0.85)
# print(all_locations)

# --- display image with green rectangles ---

img = Image.open(screen)  # load image with screenshot
draw = ImageDraw.Draw(img)  # create object for drawing

for loc in all_locations:
    #print(loc)  # it's namedtuple https://github.com/asweigart/pyscreeze/blob/master/pyscreeze/__init__.py#L133
    #x = loc.left
    #y = loc.top
    #w = loc.width
    #h = loc.height
    x, y, w, h = loc
    draw.rectangle((x, y, x + w, y + h), outline="lime", width=2)

img.save("output.png")  # save result in file (to see it later)
img.show()  # display result (it doesn't need to save)

    Result:

    pyautogui: 0.9.54
OpenCV: True

    output.png

    output.png