cmakectest

In ctest how to check for both non-zero exit code and specific string output?

I want to test that an app is both exiting with a non-zero code AND outputting a particular string. I set the properties like this:

set_tests_properties(
    myapp-test-foobar
PROPERTIES
    WILL_FAIL TRUE
    FAIL_REGULAR_EXPRESSION "^Usage: myapp"
)

But the test passes even if the app's exit code is 0. Although WILL_FAIL is necessary, otherwise the test will fail when the FRE is matched, the exit code is ignored. I searched the cmake docs for an answer to no avail.

EDIT: It turns out that ctest (I'm using v3.19.2) does not check the exit code if either PASS_REGULAR_EXPRESSION or FAIL_REGULAR_EXPRESSION is set. The pass or fail condition is met solely by the RE matching.

A nasty bug has escaped into the wild in one of my apps because of this.

EDIT: The answer suggested in comments below is for handling an app that aborts, i.e. crashes, on the error condition being tested and so is different from my case. Please see my comment below for further reasoning and solution.

Solution

  • The fix I described in my comment above does not work. As I recently discovered a mismatched FAIL_REGULAR_EXPRESSION will not cause the test to fail. See this question.

    This is because when FAIL_REGULAR_EXPRESSION and WILL_FAIL are set ctest does

    if (match FAIL_REGULAR_EXPRESSION || exit_code != 0)

    For clarity here are the other cases. When only WILL_FAIL is set, ctest does

    if (exit_code != 0)

    When PASS_REGULAR_EXPRESSION is set, ctest does

    if (match PASS_REGULAR_EXPRESSION)

    and when none of the 3 are set, ctest does

    if (exit_code == 0)

    You can test for both the correct error code and error message by abusing PASS_REGULAR_EXPRESSION as in the following

    add_test( NAME myapp-test-ktx2-in-exit-code
    COMMAND myapp -o foo infile.ktx2
    WORKING_DIRECTORY ${MY_WORK_DIR}
)
set_tests_properties(
    myapp-test-ktx2-in-exit-code
PROPERTIES
    WILL_FAIL TRUE
)
add_test( NAME myapp-test-ktx2-in
    COMMAND myapp -o foo input.ktx2
    WORKING_DIRECTORY ${MY_WORK_DIR}
)
set_tests_properties(
    myapp-test-ktx2-in
PROPERTIES
    PASS_REGULAR_EXPRESSION ".* is not a KTX v1 file."
)

    If both tests pass we are guaranteed that we have a non-zero exit code and the error message matches PASS_REGULAR_EXPRESSION.

    If you want to test both for the correct PASS_REGULAR_EXPRESSION and exit_code == 0 you will similarly need two tests: one to test PASS_REGULAR_EXPRESSION and one without it set.