mavensurefire

Maven + Surefire return code is 0 on failed tests


I have a project with tests split in unit and integration phases. I have it running buildbot and the problem is that even in tests are failing maven return code is 0 so buildbot build is succesful.

This is the result of mvn integration-test:

Results :

Tests in error: 
 Info about failed tests

Tests run: 5, Failures: 0, Errors: 5, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 19 seconds
[INFO] Finished at: Tue Feb 12 09:43:53 UTC 2013
[INFO] Final Memory: 36M/97M
[INFO] ------------------------------------------------------------------------

$ echo $?
0

Result for mvn install is the same without the build successful part Results :

Tests in error: 
  Info about failed tests

Tests run: 5, Failures: 0, Errors: 5, Skipped: 0

$ echo $?
0

Surefire configuration is like this:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.13</version>
  <configuration>
    <printSummary>true</printSummary>
    <excludedGroups>com.testlib.IntegrationTest</excludedGroups>
  </configuration>
  <executions>
    <execution>
    <id>unit-tests</id>
    <phase>test</phase>
    <goals>
      <goal>test</goal>
    </goals>
    <configuration>
          <excludedGroups>com.testlib.IntegrationTest</excludedGroups>
    </configuration>
    </execution>
    <execution>
    <id>integration-tests</id>
    <phase>integration-test</phase>
    <goals>
      <goal>test</goal>
    </goals>
    <configuration>
      <includes>
            <groups>com.testlib.IntegrationTest</groups>
      </includes>
        </configuration>
    </execution>
  </executions>
</plugin>

I've reading other threads about maven return codes but in theory related bugs should be fixed in my maven version (Apache Maven 2.2.1 (rdebian-8))

Is there any way to change this behaviour?

Update: As suggested I tried with surefire:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.13</version>
    <dependencies>
        <dependency>
        <groupId>org.apache.maven.surefire</groupId>
        <artifactId>surefire-junit47</artifactId>
        <version>2.13</version>
        </dependency>
    </dependencies>
    <configuration>
        <groups>com.testlib.IntegrationTest</groups>
    </configuration>
    <executions>
        <execution>
            <goals>
            <goal>integration-test</goal>
            <goal>verify</goal>
            </goals>
            <configuration>
                <includes>
                  <include>**/*.class</include>
                </includes>
            </configuration>
        </execution>
    </executions>

I need the surefire-junit to avoid initialization errors.


Solution

  • I managed to have it working with two different configurations, using groups and using folders using just surefire

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.13</version>
        <configuration><excludedGroups>com.testlib.IntegrationTest</excludedGroups>
        </configuration>
        <executions>
            <execution>
                <id>integration-test</id>
                <phase>integration-test</phase>
                <goals>
                    <goal>test</goal>
                </goals>
                <configuration>
                  <skip>false</skip>
                          <excludedGroups>com.biicode.testlib.UnitTest</excludedGroups>
                          <groups>com.testlib.IntegrationTest</groups>
                </configuration>
            </execution>
        </executions>
    
     <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
            <excludes>
                <exclude>**/integration/*.java</exclude>
            </excludes>
        </configuration>
        <executions>
        <execution>
            <id>integration-test</id>
            <phase>integration-test</phase>
            <goals>
                <goal>test</goal>
            </goals>
            <configuration>
            <skip>false</skip>
            <excludes>
                <exclude>none</exclude>
            </excludes>
            <includes>
                <include>**/integration/*.java</include>
            </includes>
        </configuration>
      </execution>
      </executions>
    </plugin>