Why test case is stopped even if FailureHandling.CONTINUE_ON_FAILURE is used

I don’t understand why test case is stopped whereas I use FailureHandling_CONTINUE_ON_FAILURE into caller.

Code into caller:

...
switch (testCaseName) {
    case testCaseName:
        WebUI.callTestCase(findTestCase(testCaseName), param, FailureHandling.CONTINUE_ON_FAILURE)
        break
}
...

Error into callee:

This error occured into test case called with WebUI.callTestCase(...)

Caused by: com.kms.katalon.core.webui.exception.WebElementNotFoundException: Web element with id: ‘XXX’ located by ‘By.xpath: XXX’]’ not found

But, if I use FailureHandling_CONTINUE_ON_FAILURE callee must be stopped, but not caller right?

Thanks for help

Solution

I was able to partially reproduce the issue. I wrote the following two test cases:

TC1: Caller

println ">>>> start caller"
if(!WebUI.callTestCase(findTestCase("Callee"), null, FailureHandling.OPTIONAL)){
    println ">>>> success!"
}
println ">>>> end caller"

TC2: Callee

println ">>>> start callee"
assert 0

This is my console output with FailureHandling.OPTIONAL:

2019-07-03 12:43:41.851 DEBUG testcase.Caller                          - 1: println(">>>> start caller")
>>>> start caller
2019-07-03 12:43:41.855 DEBUG testcase.Caller                          - 2: if (!(callTestCase(findTestCase("Callee"), null, OPTIONAL)))
2019-07-03 12:43:41.934 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2019-07-03 12:43:41.934 INFO  c.k.katalon.core.main.TestCaseExecutor   - CALL Test Cases/Callee
2019-07-03 12:43:42.086 DEBUG testcase.Callee                          - 1: println(">>>> start callee")
>>>> start callee
2019-07-03 12:43:42.087 DEBUG testcase.Callee                          - 2: assert 0
2019-07-03 12:43:42.095 ERROR c.k.katalon.core.main.TestCaseExecutor   - ❌ Test Cases/Callee FAILED.
Reason:
Assertion failed: 

assert 0

    at Callee.run(Callee:18)
    at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
    ....

2019-07-03 12:43:42.104 INFO  c.k.katalon.core.main.TestCaseExecutor   - END CALL Test Cases/Callee
2019-07-03 12:43:42.104 INFO  c.k.katalon.core.main.TestCaseExecutor   - --------------------
2019-07-03 12:43:42.109 WARN  c.k.k.core.keyword.internal.KeywordMain  - Unable to call Test Case 'Test Cases/Callee' (Root cause: com.kms.katalon.core.exception.StepFailedException: Call Test Case 'Test Cases/Callee' failed
    at com.kms.katalon.core.keyword.builtin.CallTestCaseKeyword$_callTestCase_closure1.doCall(CallTestCaseKeyword.groovy:63)
    ....
Caused by: Assertion failed: 

assert 0

    at Callee.run(Callee:18)
    at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
    ...
)
2019-07-03 12:43:42.110 DEBUG testcase.Caller                          - 1: println(">>>> success!")
>>>> success!
2019-07-03 12:43:42.111 DEBUG testcase.Caller                          - 3: println(">>>> end caller")
>>>> end caller
2019-07-03 12:43:43.067 INFO  c.k.katalon.core.main.TestCaseExecutor   - END Test Cases/Caller

So, TC2 is failed, but WebUI.callTestCase() throws an exception that is interpreted as WARN in the logs, so TC1 is passed.

When I switch to FailureHandling.CONTINUE_ON_FAILURE, that is interpreted as ERROR in the logs and both TC1 and TC2 are failed. More on failure handling here.

I believe this is a bug in Katalon Studio. But, I think you can use FailureHandling.OPTIONAL as a workaround to get the desired result.