cmakectest

ctest: Disable a set of labeled tests by default - but run them when explicitly targeted


I'd like to disable a set of tests by default, but be able to run them when explicitly targeted.

E.g. suppose I have a project with lots of fast-running unit tests that have been added via a add_test(NAME SomeNameHere COMMAND SomeCommandHere) command. And suppose further that there are also a bunch of long-running integration tests that should be excluded from the default ctest run, But it should be possible to run all integration tests with a single command when explicity targeted.

As a concrete example:

Here's a my attempt to achieve this.

cmake_minimum_required (VERSION 3.1)
project (HOW_TO_RUN_DISABLED_TESTS NONE)
enable_testing()

# Add a fast running test
# This test should be run when a user runs: 'ctest'
add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')

# Add a long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
set_tests_properties(FooIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(FooIntegrationTest PROPERTIES LABELS my_integration_tests)

# Add another long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
set_tests_properties(BarIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(BarIntegrationTest PROPERTIES LABELS my_integration_tests)

Now, let's run this via

$ mkdir build
$ cd build/
$ cmake ..
$ ctest -V

As we can see in the output (posted below) ctest did exclude the integration tests (great!).

1: Test command: /bin/echo "'I" "am" "always" "enabled'"
1: Test timeout computed to be: 10000000
1: 'I am always enabled'
1/3 Test #1: TestThatShouldAlwaysRun ..........   Passed    0.00 sec
test 2
    Start 2: FooIntegrationTest
2/3 Test #2: FooIntegrationTest ...............***Not Run (Disabled)   0.00 sec
test 3
    Start 3: BarIntegrationTest
3/3 Test #3: BarIntegrationTest ...............***Not Run (Disabled)   0.00 sec

100% tests passed, 0 tests failed out of 1

Label Time Summary:
my_integration_tests    =   0.00 sec*proc (2 tests)

Total Test time (real) =   0.05 sec

The following tests did not run:
      2 - FooIntegrationTest (Disabled)
      3 - BarIntegrationTest (Disabled)

But, I can not figure out how I can now, through ctest, run tests labeled with my_integration_tests.

ctest --label-regex my_integration_tests
Test project /path/to/code/example_repo/build
    Start 2: FooIntegrationTest
1/2 Test #2: FooIntegrationTest ...............***Not Run (Disabled)   0.00 sec
    Start 3: BarIntegrationTest
2/2 Test #3: BarIntegrationTest ...............***Not Run (Disabled)   0.00 sec
No tests were found!!!

Is there a way to run disabled tests when explicitly targeted?

I have explored other ways like


Solution

  • I believe a simpler way to do this is just via a cmake configuration option.

    # cat CMakeLists.txt 
    cmake_minimum_required (VERSION 3.1)
    project (HOW_TO_RUN_DISABLED_TESTS NONE)
    enable_testing()
    add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')
    if(BUILD_INTEGRATION_TESTING)
       add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
       add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
    endif()
    

    This is simple and also is more flexible, ie. enables addition of test dependent targets to be build (and setup (and ctest fixtures to be used)) for those tests. (I usually in projects see variables named like SOMETHING_BUILD_TESTING so it's similar to BUILD_TESTING cmake variable.)

    Another idea I have is to use an environment variable and wrap the test in a script with SKIP_RETURN_CODE (or even without SKIP_RETURN_CODE and just return success if integration tests shouldn't be run):

    add_test(NAME FooIntegrationTest 
            COMMAND sh -c "if [ \"\${RUN_INTEGRATION:-}\" ]; then echo 'Foo integration test'; else exit 127; fi"
            VERBATIM)
    set_tests_properties(FooIntegrationTest PROPERTIES 
            SKIP_RETURN_CODE 127)
    

    then:

    $ ctest -V
    ....
    2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
    2: Test timeout computed to be: 10000000
    2/3 Test #2: FooIntegrationTest ...............***Skipped   0.01 sec
    ....
    The following tests did not run:
          2 - FooIntegrationTest (Skipped)
    

    but:

    $ RUN_INTEGRATION=TRUE ctest -V
    ...
    2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
    2: Test timeout computed to be: 10000000
    2: Foo integration test
    2/3 Test #2: FooIntegrationTest ...............   Passed    0.01 sec
    ...