java maven junit5 cucumber-java cucumber-junit

How to run unit tests (non-cucumber) and Cucumber tests (Junit5 based) separately with maven in Spring Boot 3 project?

Problem: When using newer version of Cucumber & Junit5 with cucumber-junit-platform-engine, mvn test runs both unit tests and integration tests. We can no longer run them separately, and I'm looking for a way to fix this.

We had unit tests running om mvn test and cucumber integration tests on mvn verify with the surefire and failsafe plugin. After update to Junit5 and Cucumber 7.15.0 we now use cucumber-junit-platform-engine for maven to discover the cucumber tests. The problem is, that we no longer have the option to run these tests separately. Both mvn test and mvn verify run all the tests (unit tests and cucumber tests) together.

How can we make a distinction between running the unit tests on mvn test and the cucumber integration tests on mvn verify?

Project structure:

- src
   |-integration-test
      |-resources with feature files, and application.yml for tests
      |-java
         |-com.project.package.structure
            |- scenariosteps
            |- CucumberIT.java with @Suite annotation
            |- IntegrationTestConfig with @TestConfiguration annotation
            |- SpringEntry.java
   |-Spring boot 3 project
   |-test
      |-java
         |-com.project.package.structure
            |-our junit5 + mockito unit tests

Dependencies (all spring boot 3 project dependencies omitted)

Maven surefire plugin 3.2.5
Maven failsafe plugin 3.2.1

                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-java</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-junit</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-spring</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>org.junit.jupiter</groupId>
                    <artifactId>junit-jupiter</artifactId>
                    <version>5.10.1</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>org.junit.platform</groupId>
                    <artifactId>junit-platform-suite</artifactId>
                    <version>1.10.1</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-junit-platform-engine</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>

I've been looking through docs + stackoverflow on how to do this. When we were using org.apache.maven.surefire.junitplatform.JUnitPlatformProvider this worked fine, but it looks like io.cucumber.junit.platform.engine.DiscoverySelectorResolver overrides everything:

I get this warning when running mvn test, before unit tests and Cucumber tests are executed:

io.cucumber.junit.platform.engine.DiscoverySelectorResolver warnWhenCucumberFeaturesPropertyIsUsed WARNING: Discovering tests using the cucumber.features property. Other discovery selectors are ignored! Please request/upvote/sponsor/ect better support for JUnit 5 discovery selectors. See: https://github.com/cucumber/cucumber-jvm/pull/2498

When I remove cucumber-junit-platform-engine from the dependencies, the following is used. But this fails to pick up the Cucumber tests. Looks like switching back to JUnitPlatformProvider is not an option, since this is no longer compatible with Cucumber tests.

Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider

My surefire/failsafe config:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven-surefire-plugin.version}</version>
                <configuration>
                    <skipTests>${unit-tests.skip}</skipTests>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>${maven-failsafe-plugin.version}</version>
                <dependencies>
                    <dependency>
                        <groupId>io.cucumber</groupId>
                        <artifactId>cucumber-junit-platform-engine</artifactId>
                        <version>${cucumber.version}</version>
                    </dependency>
                </dependencies>
                <configuration>
                    <properties>
                        <configurationParameters>
                            cucumber.junit-platform.naming-strategy=long
                        </configurationParameters>
                    </properties>
                </configuration>
                <executions>
                    <execution>
                        <id>integration-tests</id>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <skipTests>${integration-tests.skip}</skipTests>
                            <runOrder>alphabetical</runOrder>
                            <parallel>none</parallel>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

Solution

This is what I endend up doing:

To fix the problem where mvn test ran both unit and cucumber integration tests, and we got the io.cucumber.junit.platform.engine.DiscoverySelectorResolver warnWhenCucumberFeaturesPropertyIsUsed warning when running mvn test/verify I used @M.P. Korstanje's solution:

Removed the cucumber.features from junit-platform.properties. And for CucumberIT.java:

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
// Worked the same as the line above: @ConfigurationParameter(key = FEATURES_PROPERTY_NAME, value = "src/integration-test/resources/features")
public class CucumberIT {
}

Dependencies:

                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-java</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-junit</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-spring</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>org.junit.jupiter</groupId>
                    <artifactId>junit-jupiter</artifactId>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>org.junit.platform</groupId>
                    <artifactId>junit-platform-suite</artifactId>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>io.cucumber</groupId>
                    <artifactId>cucumber-junit-platform-engine</artifactId>
                    <version>7.15.0</version>
                    <scope>test</scope>
                </dependency>

My surefire and failsafe:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
                <configuration>
                    <skipTests>${unit-tests.skip}</skipTests>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>3.1.2</version>
                <configuration>
                    <properties>
                        <configurationParameters>
                            cucumber.junit-platform.naming-strategy=long
                        </configurationParameters>
                    </properties>
                </configuration>
                <executions>
                    <execution>
                        <id>integration-tests</id>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <skipTests>${integration-tests.skip}</skipTests>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

The final problem was Failsafe kept running the integration tests twice, this turned out to be a bit of old config <runOrder>alphabetical</runOrder><parallel>none</parallel> just below <skipTests>${integration-tests.skip}</skipTests> in the failsafe plugin.