I'd like to disable a set of tests by default, but be able to run them when explicitly targeted.
E.g. suppose I have a project with lots of fast-running unit tests that have been added via a add_test(NAME SomeNameHere COMMAND SomeCommandHere)
command. And suppose further that there are also a bunch of long-running integration tests that should be excluded from the default ctest
run, But it should be possible to run all integration tests with a single command when explicity targeted.
As a concrete example:
TestThatShouldAlwaysRun
- I'd like this to run when a user runs $ ctest
FooIntegrationTest
- I'd like this to run only when a user runs $ ctest --some-option-here integration
BarIntegrationTest
- I'd like this to run only when a user runs $ ctest --some-option-here integration
Here's a my attempt to achieve this.
cmake_minimum_required (VERSION 3.1)
project (HOW_TO_RUN_DISABLED_TESTS NONE)
enable_testing()
# Add a fast running test
# This test should be run when a user runs: 'ctest'
add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')
# Add a long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
set_tests_properties(FooIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(FooIntegrationTest PROPERTIES LABELS my_integration_tests)
# Add another long-running integration test
# This test should be run when a user runs: 'ctest --label-regex my_integration_tests'
add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
set_tests_properties(BarIntegrationTest PROPERTIES DISABLED True)
set_tests_properties(BarIntegrationTest PROPERTIES LABELS my_integration_tests)
Now, let's run this via
$ mkdir build
$ cd build/
$ cmake ..
$ ctest -V
As we can see in the output (posted below) ctest
did exclude the integration tests (great!).
1: Test command: /bin/echo "'I" "am" "always" "enabled'"
1: Test timeout computed to be: 10000000
1: 'I am always enabled'
1/3 Test #1: TestThatShouldAlwaysRun .......... Passed 0.00 sec
test 2
Start 2: FooIntegrationTest
2/3 Test #2: FooIntegrationTest ...............***Not Run (Disabled) 0.00 sec
test 3
Start 3: BarIntegrationTest
3/3 Test #3: BarIntegrationTest ...............***Not Run (Disabled) 0.00 sec
100% tests passed, 0 tests failed out of 1
Label Time Summary:
my_integration_tests = 0.00 sec*proc (2 tests)
Total Test time (real) = 0.05 sec
The following tests did not run:
2 - FooIntegrationTest (Disabled)
3 - BarIntegrationTest (Disabled)
But, I can not figure out how I can now, through ctest
, run tests labeled with my_integration_tests
.
ctest --label-regex my_integration_tests
Test project /path/to/code/example_repo/build
Start 2: FooIntegrationTest
1/2 Test #2: FooIntegrationTest ...............***Not Run (Disabled) 0.00 sec
Start 3: BarIntegrationTest
2/2 Test #3: BarIntegrationTest ...............***Not Run (Disabled) 0.00 sec
No tests were found!!!
Is there a way to run disabled tests when explicitly targeted?
I have explored other ways like
ctest --label-exclude my_integration_tests
. Which isn't great. That puts a burdon on everyone to remember this. I'd like the options that's used the most (running only the fast-running tests) to be runnable via ctest
.CONFIGURATIONS my_integration_tests
to the add_test
call. But I think that way I am semantically misusing CONFIGURATIONS
. Also this doesn't allow me to run only the integration test when targeted.add_custom_target
. Also here I think I am misusing the API. add_custom_target
is meant for custom targets, not as a way to run test, right?I believe a simpler way to do this is just via a cmake configuration option.
# cat CMakeLists.txt
cmake_minimum_required (VERSION 3.1)
project (HOW_TO_RUN_DISABLED_TESTS NONE)
enable_testing()
add_test(NAME TestThatShouldAlwaysRun COMMAND echo 'I am always enabled')
if(BUILD_INTEGRATION_TESTING)
add_test(NAME FooIntegrationTest COMMAND echo 'Foo integration test')
add_test(NAME BarIntegrationTest COMMAND echo 'Bar integration test')
endif()
This is simple and also is more flexible, ie. enables addition of test dependent targets to be build (and setup (and ctest fixtures to be used)) for those tests. (I usually in projects see variables named like SOMETHING_BUILD_TESTING
so it's similar to BUILD_TESTING
cmake variable.)
Another idea I have is to use an environment variable and wrap the test in a script with SKIP_RETURN_CODE
(or even without SKIP_RETURN_CODE
and just return success if integration tests shouldn't be run):
add_test(NAME FooIntegrationTest
COMMAND sh -c "if [ \"\${RUN_INTEGRATION:-}\" ]; then echo 'Foo integration test'; else exit 127; fi"
VERBATIM)
set_tests_properties(FooIntegrationTest PROPERTIES
SKIP_RETURN_CODE 127)
then:
$ ctest -V
....
2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
2: Test timeout computed to be: 10000000
2/3 Test #2: FooIntegrationTest ...............***Skipped 0.01 sec
....
The following tests did not run:
2 - FooIntegrationTest (Skipped)
but:
$ RUN_INTEGRATION=TRUE ctest -V
...
2: Test command: /usr/bin/sh "-c" "if [ "${RUN_INTEGRATION:-}" ]; then echo 'Foo integration test'; else exit 127; fi" "VERBATIM"
2: Test timeout computed to be: 10000000
2: Foo integration test
2/3 Test #2: FooIntegrationTest ............... Passed 0.01 sec
...