scalaunit-testingscala.jsutest

How to test objects or other single resources / disable parallel testing with uTest

https://github.com/lihaoyi/utest

As said in the documentation uTest runs tests parallel. Test cases inside one test suite can be run sequentially if used 

import utest.ExecutionContext.RunNow

but sbt runs different test suites always parallel. https://github.com/lihaoyi/utest#parallel-testing

This is usually good for performance but when testing objects or other "single resources" this can cause problems. If same object is tested or used in several test suites, these parallel tests may disturb each other.

Logging is a good example: Logging may use a commom factory object and in some cases this object can contain state like default logging level or data structure for log instances etc. Logging is used in several SW modules / in several test suites and if one of those test suites tests the logging itself then the other test suites may cause random false results in the logging test suite. Database is another example of "single resource".

Is there any way to limit the parallel running of uTest test suites so that the user can decide which test suites are always run sequentially and which test suites can be run parallel?

E.g: Run test suites: com.XXX.yyy.MyTestSuiteA, com.XXX.yyy.MyTestSuiteB and com.XXX.yyy.MyTestSuiteC sequentially (in this order) and the rest parallel (D-Z).

sbt "script" is not a good solution

sbt testOnly com.XXX.yyy.MyTestSuiteA
sbt testOnly com.XXX.yyy.MyTestSuiteB
sbt testOnly com.XXX.yyy.MyTestSuiteC
sbt testOnly com.XXX.yyy.MyTestSuiteD
sbt testOnly com.XXX.yyy.MyTestSuiteE
sbt testOnly com.XXX.yyy.MyTestSuiteF
...

because then nothing is run parallel and I must list all test suites. Something like this might be good:

sbt test -- sequentially com.XXX.yyy.MyTestSuiteA, com.XXX.yyy.MyTestSuiteB, com.XXX.yyy.MyTestSuiteC

meaning that the rest can be run parallel and in whatever order. Or

sbt test -- sequentially

meaning that all test suites are run sequentially

Solution

  • Here is a quick fix which you can add to your project build file, which can be used for complete sequential execution.

    parallelExecution in Test := false

    Quoting the actual documentation.

    Disable Parallel Execution of Tests¶ By default, sbt runs all tasks in parallel. Because each test is mapped to a task, tests are also run in parallel by default. To make tests within a given project execute serially:

    scala parallelExecution in Test := false Test can be replaced with IntegrationTest to only execute integration tests serially. Note that tests from different projects may still execute concurrently.

    For other use case, I would suggest try tagging tests and running the tagged tests exclusively.

    Tags.exclusive(Benchmark)

    Refer to this doc for more information here.