CTest: Run single flakey test in parallel until it fails

I would like to run a single test in parallel until it fails with CTest. How can I achieve this?

We can assume that the test fails around every 10 000th time, and that there is 100+ cores on the machine for this question.

Further, we can assume that the test itself has some internal waiting and even on a single core you could run 10 tests in parallel, making the total run-time reduction potentially 1000x.

The problem here is that the normal way of running CTest in parallel is to use:

ctest . --repeat until-fail:100 -j100 -R tests_

This works great when running hundreds or thousands of tests. When you want to run a single test in parallel, I have not been able to do that.

ctest . --repeat until-fail:10000 -j100 -R a_specific_test

This test is then run serially, 10 000 times, which takes around 3 hours. It is also impossible to recreate the flakiness, as many flaky tests require the system to be under a certain load.

Solution

Create the same test a 1000 times

I'm not sure how to solve this elegantly. But here is a possible solution for you.

Essentially make the same test X amount of times and just regex for it.

foreach(loopVar RANGE 0 1000 1)
    add_test(NAME flaky-${loopVar} COMMAND ...)
endforeach()

The main thing you want to test is the COMMAND which is constant for all flaky tests.

Then just run ctest with flaky for the regex.

> ctest ... -R flaky -j 100
          Start    1: flaky-0
   1/1001 Test    #1: flaky-0 ..........................   Passed    0.00 sec
          Start    2: flaky-1
   2/1001 Test    #2: flaky-1 ..........................   Passed    0.00 sec
          Start    3: flaky-2
   3/1001 Test    #3: flaky-2 ..........................   Passed    0.00 sec
          Start    4: flaky-3

....

 996/1001 Test  #713: flaky-712 ........................   Passed    0.01 sec
 997/1001 Test  #686: flaky-685 ........................   Passed    0.01 sec
 998/1001 Test  #803: flaky-802 ........................   Passed    0.01 sec
 999/1001 Test  #554: flaky-553 ........................   Passed    0.01 sec
1000/1001 Test  #464: flaky-463 ........................   Passed    0.01 sec
1001/1001 Test  #216: flaky-215 ........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1001