javamavenjunitjunit4maven-failsafe-plugin

run suites in parallel using maven failsafe

I am trying to run my tests in parallel, and I have a use-case different from all others that I have been able to find.

My tests are laid out pretty straight-forward, something like the following:

src/test/java
+-features.areaA
| +-SomeStory.java
| +-AnotherStory.java
| ...
+-features.areaB
| +-DifferentStory.java
| +-OtherStory.java
| ...
...

The tests are written using , which is a wrapper for , and the test manager is .

Each "area" represents some discreet area of the application under test. Tests within one area cannot run in parallel as they would clobber the data they are using. However, tests between different areas can certainly run in parallel as there are no collisions.

I tried to configure my according to the documentation. Using parallel=suites and any one of threadCount=4, threadCountSuites=4, or useUnlimitedThreads=true, results in only one test being run at a time.

Is my understanding of "suites" wrong in the context of Failsafe plugin? Is it possible to parallelize tests so that entire packages are fed into VM threads one at a time, but classes within one package run sequentially?

Update:

Solution

  • has no notion of "packages". The notion of "suites" is related to Suite.

    To solve my problem above, I did the following:

    In each "features.area" I created a "TestSuiteStub":

    import org.junit.runner.RunWith;
import org.junit.runners.Suite;

/**
 * This will be processed by groovy-maven-plugin.
 */
@RunWith(Suite.class)
@Suite.SuiteClasses({
/* add test Stories after this ANCHOR */
})
public class TestSuiteStub {
    /* Suite holder */
}

    I then used the groovy-maven-plugin with execute goal and the following script:

    // find all TestSuiteStub.java
new File("${project.build.testSourceDirectory}/features").  // exploits the "complication" in Maven interpolation and GStrings
  traverse(type: groovy.io.FileType.FILES, nameFilter: ~/TestSuiteStub\.java/) { stub ->
    println 'Using: ' + stub

    // in the same dir as TestSuiteStub.java, find all *Story.java
    def stories = new StringBuilder()
    new File(stub.parent).
      eachFileMatch(groovy.io.FileType.FILES, ~/.*Story\.java/) { story ->
        stories.append story.name.replace('java', 'class')
        stories.append ', ' // will leave a comma at end of list, but javac seems to be ok with that
      }
    println 'Found: ' + stories

    // write out TestSuite.java
    def suite = new File(stub.parent + '/TestSuite.java')
    suite.delete()
    println 'Writing: ' + suite
    stub.eachLine() { line ->

      if(line.contains('Stub'))
        suite.append line.replace('TestSuiteStub', 'TestSuite') + System.getProperty('line.separator')
      else
        suite.append line + System.getProperty('line.separator')

      if(line.contains('ANCHOR'))
        suite.append stories + System.getProperty('line.separator')
    }
  }

    This will process each of the TestSuiteStub and generate a TestSuite. So after running this (you can use mvn test-compile to run this, without running your tests), I have something like:

    src/test/java
+-features.areaA
| +-SomeStory.java
| +-AnotherStory.java
| ...
| +-TestSuiteStub.java
| +-TestSuite.java
+-features.areaB
| +-DifferentStory.java
| +-OtherStory.java
| ...
| +-TestSuiteStub.java
| +-TestSuite.java
...

    where the first TestSuite will look something like:

    import org.junit.runner.RunWith;
import org.junit.runners.Suite;

/**
 * This class will be processed by groovy-maven-plugin.
 */
@RunWith(Suite.class)
@Suite.SuiteClasses({
/* add test Stories after this ANCHOR */
SomeStory.class, AnotherStory.class,
})
public class TestSuite {
    /* Suite holder */
}

    To make pick this up, I configured it like this:

    <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>${failsafe.plugin.version}</version>
  <configuration>
    <includes>
      <include>features.*.*Suite</include>
    </includes>
    <parallel>suites</parallel>
    <threadCountSuites>4</threadCountSuites>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>integration-test</goal>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

    Lastly I also added *Suite.java to my .gitignore.