[SOLVED] Squish test functions always pass when used with squishtest module

Squish test functions always pass when used with squishtest module

We are using Squish for Qt 6.6.2 on Windows 10 with Python 3.8.7 and running our tests using squishtest module with Robot Framework 4.0.1.

We are having an issue with the test functions provided by the Squish API where any verifications done with such a call (for example squishtest.test.imagePresent) will Pass. The issue itself was quite simple to pinpoint to the fact that although the verification failed, the function call itself was passing without raising exceptions. This can also be verified from the report provided by the squishrunner where we have <scriptedVerificationResult type="FAIL" time="--"> on the passed execution.

The question is, can we in any way get the actual verification result passed to the Robot so we can fail the test accordingly? Preferrably in real time rather than parsing the report afterwards.

In Squish this works perfectly fine

def main():
    startApplication("AUT")
    snooze(2)
    test.imagePresent("image.png", {"tolerant": True, "threshold": 85}, 
    waitForObjectExists(names.sceneContainer_GraphWidget))

but with Robot this is always passing

# In testSuite.robot

*** Settings ***
Library     MySquishLib

*** Test Cases ***
Test Image
    Start AUT
    Verify Image    image.png    {"tolerant": True, "threshold": 85}    names.sceneContainer_GraphWidget

# In MySquishLib.py

import squishtest
import names

def start_aut():
    squishtest.startApplication("AUT")

def verify_image(imageFile, imageParams, imageArea):
    squishtest.test.imagePresent(imageFile, imageParams, imageArea)

Solution

The test functions are not supposed to raise exceptions during execution in order to allow test to continue even if a single VP was failed. The function does however return a boolean value just as expected. By using

def verify_image(imageFile, imageParams, imageArea):
    if not squishtest.test.imagePresent(imageFile, imageParams, imageArea):
        raise Exception("Image was not found")

I'm able to fail the Robot test without any issues.