maven maven-3 integration-testing pom.xml maven-invoker-plugin

Maven Invoker Plugin not detecting failed test

As part of a root parent pom project, several integration tests have been added to test it on sample projects.

The structure of the project folder is as following:

-root-maven-parent-project
    |- src
    |   |-it
    |      |-sample-project-test1
    |      |-sample-project-test2
    |      |-sample-project-test3
    |      |-settings.xml
    |- pom.xml

The main issue is: although the build of sample-project-test2 is wrongly failing (it should not), the build is SUCCESSFUL for the Invoker plugin and the overall build does not fail.

Here is the concerned maven-invoker-plugin configuration:

<profile>
    <id>it-tests</id>
    <build>
        <plugins>
            <!-- Integration tests configuration -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-invoker-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <streamLogs>true</streamLogs>
                    <goals>
                        <goal>clean</goal>
                        <goal>generate-sources</goal>
                    </goals>
                    <settingsFile>src/it/settings.xml</settingsFile>
                    <failIfNoProjects>true</failIfNoProjects>
                </configuration>
                <executions>
                    <execution>
                        <id>integration-test-release</id>
                        <goals>
                            <goal>install</goal>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <cloneProjectsTo>${project.build.directory}/its/sample-project-test1</cloneProjectsTo>
                            <pom>src/it/sample-project-test1/pom.xml</pom>
                            <properties>
                                <scmBranch>release-something</scmBranch>
                            </properties>
                        </configuration>
                    </execution>
                    <execution>
                        <id>integration-test-hotfix</id>
                        <goals>
                            <goal>install</goal>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <cloneProjectsTo>${project.build.directory}/its/sample-project-test2</cloneProjectsTo>
                            <pom>src/it/sample-project-test2/pom.xml</pom>
                            <properties>
                                <scmBranch>hotfix-something</scmBranch>
                            </properties>
                        </configuration>
                    </execution>
                    <execution>
                        <id>integration-test-master</id>
                        <goals>
                            <goal>install</goal>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <cloneProjectsTo>${project.build.directory}/its/sample-project-test3</cloneProjectsTo>
                            <pom>src/it/sample-project-test3/pom.xml</pom>
                            <properties>
                                <scmBranch>master</scmBranch>
                            </properties>
                        </configuration>
                    </execution>
            </plugin>
        </plugins>
    </build>
</profile>

As you can see, multiple executions are configured because each execution would need its own properties. Each execution is also pointing at its own integration test project and pom.

The build is clearly failing for a specific execution:

[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] BUILD FAILURE
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] Total time: 2.337 s
[INFO] [INFO] Finished at: 2017-07-04T17:35:49+02:00
[INFO] [INFO] Final Memory: 12M/220M
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (enforce-snapshot-management) on project cmp-sample-project-test2: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[INFO] [ERROR] 
[INFO] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[INFO] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
[INFO] [ERROR] 
[INFO] [ERROR] For more information about the errors and possible solutions, please read the following articles:
[INFO] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[INFO]           pom.xml .......................................... FAILED (4.1 s)
[INFO]   The build exited with code 1. See C:\data\git-repositories\root-maven-parent\target\its\sample-project-test2\build.log for details.

However, at the bottom of the build we see that the verify goal of the maven-invoker-plugin aggregated the results, flagged the concerned test as Passed and made the build SUCCESS:

[INFO] 
[INFO] --- maven-invoker-plugin:3.0.0:verify (integration-test-release) @ root-maven-parent ---
[INFO] -------------------------------------------------
[INFO] Build Summary:
[INFO]   Passed: 1, Failed: 0, Errors: 0, Skipped: 0
[INFO] -------------------------------------------------
[INFO] 
[INFO] --- maven-invoker-plugin:3.0.0:verify (integration-test-hotfix) @ root-maven-parent ---
[INFO] -------------------------------------------------
[INFO] Build Summary:
[INFO]   Passed: 1, Failed: 0, Errors: 0, Skipped: 0
[INFO] -------------------------------------------------
[INFO] 
[INFO] --- maven-invoker-plugin:3.0.0:verify (integration-test-master) @ root-maven-parent ---
[INFO] -------------------------------------------------
[INFO] Build Summary:
[INFO]   Passed: 1, Failed: 0, Errors: 0, Skipped: 0
[INFO] -------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Moreover, by only running the failing test from command line as:

mvn invoker:integration-test@integration-test-hotfix invoker:verify -Pit-tests

The sub-build of the test project fails, the output is correctly marked as Failed in the test summary, and the build is correctly ending with FAILURE.

Question: why when executing multiple integration tests using the maven-invoker-plugin, although a test is failed, it is marked as Passed in the test summary and the build does not fail, while running only the isolated test everything fails correctly?

Note: no invoker property file is used.

Solution

Issue solved with the following explanation, although I think something could be improved in the behavior of the plugin (see below).

The whole maven-invoker-plugin was reduced to the following configuration:

<profile>
    <id>it-tests</id>
    <build>
        <plugins>
            <!-- Integration tests configuration -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-invoker-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <streamLogs>true</streamLogs>
                    <goals>
                        <goal>clean</goal>
                        <goal>generate-sources</goal>
                    </goals>
                    <settingsFile>src/it/settings.xml</settingsFile>
                    <failIfNoProjects>true</failIfNoProjects>
                    <cloneProjectsTo>${project.build.directory}/its</cloneProjectsTo>
                </configuration>
                <executions>
                    <execution>
                        <id>integration-test-release</id>
                        <goals>
                            <goal>install</goal>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</profile>

Basically: only one plugin execution, instead of an execution per test, which indeed was verbose and non scalable, but forced by the need of having different values for the same property in each integration test. Apparently, this is not possible via pom configuration and only achievable - unless I am not mistaken - via a test.properties file. Hence, as a complement to the configuration above, I added in each and every integration test project folder a test.properties file with the following content e.g.:

scmBranch=master

De facto replacing what in the pom.xml file was (as part of an execution of the maven-invoker-plugin:

<properties>
    <scmBranch>master</scmBranch>
</properties>

This mechanism (single execution of the plugin + test properties file per test folder) fixed the issue, allowing the build to have multiple integration tests each with its own different value for the same property. Hopefully this solution may help troubleshooting similar issues.

Here is the final result from the build correctly aggregating tests and effectively respecting their sub-build output (while before the build was generating 6 Build Summary of Passed: 1 each time, although not correct).

[INFO] --- maven-invoker-plugin:3.0.0:verify (pom-integration-test) @ root-maven-parent ---
[INFO] -------------------------------------------------
[INFO] Build Summary:
[INFO]   Passed: 6, Failed: 0, Errors: 0, Skipped: 0
[INFO] -------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

However, some questions remain:

Without using the test.properties file, how to achieve the same via pom.xml configuration? Normally, it should only be an alternative, not a mandatory and only possible solution. That's why this is rather a uncomplete feature (a bug?) to me.
Having multiple execution of the plugin results in test summaries at the end of the build which correctly follow the executions order, the number of tests executed (always 1 per execution, in this case), but apparently do not reflect the effective result of each sub-build. Why? This is rather a bug or a misbehavior of the plugin due to an unexpected usage of it, perhaps.