I am using Playwright in combination with Cucumber. All of my tests are ran in parallel through maven surefire with the JUnit 5 test runner. The issue I am having is that I have been unable to enforce a timeout for my tests, so sometimes a single test can get stuck in CI and run for hours before we notice and have to cancel the build.
I have the following junit-platform.properties
file which controls how the cucumber tests are run:
cucumber.execution.parallel.enabled=true
cucumber.execution.parallel.config.strategy=dynamic
cucumber.execution.parallel.config.dynamic.factor=1
cucumber.plugin=formatters.SummaryFormatter,pretty,junit:target/junitreport.xml,json:target/cucumber-report/cucumber.json
cucumber.execution.exclusive-resources.isolated.read-write=org.junit.platform.engine.support.hierarchical.ExclusiveResource.GLOBAL_KEY
cucumber.junit-platform.naming-strategy=long
I have the following Cucumber Runner:
@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "stepdefinitions")
@SuppressWarnings({"java:S2187", "checkstyle:AbbreviationAsWordInName"})
public class CucumberRunnerIT {
private CucumberRunnerIT() {
}
}
I have tried a variety of different solution that I've read online, however none of them worked, the tests continue to run even once the timeout value has been reached.
1: Using the @Timeout
annotation and adding this to my CucumberRunnerIT
@Timeout(value = 10, unit = TimeUnit.MINUTES)
2: Configuring a timeout in the junit-platform.properties
:
junit.jupiter.execution.timeout.default=10m
3: Configuring the timeout in the POM
:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M5</version>
<configuration>
<systemPropertyVariables>
<junit.jupiter.execution.timeout.default>10m</junit.jupiter.execution.timeout.default>
</systemPropertyVariables>
</configuration>
</plugin>
Regardless of any solution, the timeout is not respected.
The JUnit User Guide explains:
Unlike previous versions of JUnit, JUnit 5 is composed of several different modules from three different sub-projects.
JUnit 5 = JUnit Platform + JUnit Jupiter + JUnit Vintage
The JUnit Platform serves as a foundation for launching testing frameworks on the JVM. It also defines the TestEngine API for developing a testing framework that runs on the platform.
And like JUnit Jupiter and JUnit Vintage, Cucumber is an Engine implemented using the JUnit Platform.
The property junit.jupiter.execution.timeout.default=10m
configures the timeout for JUnit Jupiter. This has no effect on Cucumber. The @org.junit.jupiter.api.Timeout
annotation also does not work for the same reason.
Cucumber currently does not support time outs. They're actually quite complicated.