SBT test does not run zio tests despite inclusion of test framework ZTestFramework

Edited following comments by @GastónSchabas to include a minimal reproducible example.

I have a project in Scala 2.3.11, sbt 1.9.3, zio 2.0.15, zio-test 2.0.15.

In the project, I have a mix of scalatest tests org.scalatest.flatspec.AnyFlatSpec and ZIO Junit tests zio.test.junit.JUnitRunnableSpec.

Here is the build.sbt file:

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.11"

val zioVersion = "2.0.15"
val scalaTestVersion = "3.2.16"

libraryDependencies ++= Seq(
  "org.scalactic"     %% "scalactic"         % scalaTestVersion,
  "org.scalatest"     %% "scalatest"         % scalaTestVersion % Test,
  "dev.zio"           %% "zio"               % zioVersion,
  "dev.zio"           %% "zio-test"          % zioVersion % Test,
  "dev.zio"           %% "zio-test-sbt"      % zioVersion % Test,
  "dev.zio"           %% "zio-test-magnolia" % zioVersion % Test,
  "dev.zio"           %% "zio-test-junit"    % zioVersion % Test
)

testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework")

Here is the build.properties file:

sbt.version = 1.9.3

In src/test, I include exactly two tests, one scalatest and one ziotest, as follows:

import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers

class ScalaTest1 extends AnyFlatSpec with Matchers {
  "test" should "succeed" in {
    42 shouldEqual 42
  }
}

import zio._
import zio.test.junit.JUnitRunnableSpec
import zio.test.{Spec, TestEnvironment, assertTrue}

class ZioTest1 extends JUnitRunnableSpec {
  override def spec = suite("ZioTest1")(
    test("test1") {
      for {
        number <- ZIO.succeed(42)
      } yield {
        assertTrue(number == 42)
      }
    }
  )
}

My directory structure looks like this:

- src
  - test
    - scala
      ScalaTest1.scala
      ZioTest1.scala
- project
  build.properties
build.sbt

In SBT, if I try:

% sbt
 % sbt
[info] welcome to sbt 1.9.3 (Eclipse Adoptium Java 17.0.2)
...
sbt:zio-test-with-sbt> show testFrameworks
[info] * TestFramework(org.scalacheck.ScalaCheckFramework)
[info] * TestFramework(org.specs2.runner.Specs2Framework, org.specs2.runner.SpecsFramework)
[info] * TestFramework(org.specs.runner.SpecsFramework)
[info] * TestFramework(org.scalatest.tools.Framework, org.scalatest.tools.ScalaTestFramework)
[info] * TestFramework(com.novocode.junit.JUnitFramework)
[info] * TestFramework(munit.Framework)
[info] * TestFramework(zio.test.sbt.ZTestFramework)
[info] * TestFramework(weaver.framework.CatsEffect)
[info] * TestFramework(hedgehog.sbt.Framework)
[info] * TestFramework(zio.test.sbt.ZTestFramework)
sbt:zio-test-with-sbt>

So, ZTestFramework has been correctly added.

I can run the ZIO Junit tests easily from within Intellij.

However, when I ran sbt test, I was expecting my ZIO tests to run. Instead, only the scalatest tests ran.

 % sbt test
[info] welcome to sbt 1.9.3 (Eclipse Adoptium Java 17.0.2)
...
[info] ScalaTest1:
[info] test
[info] - should succeed
[info] Run completed in 274 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 1 s, completed Aug 9, 2023, 4:04:43 PM

Any idea why it's not picking up my zio tests? Is there any way to debug the ZTestFramework further?

Solution

The short answer is, ZioTest1 have to be changed from class to object. As it is detailed in Writing Our First ZIO Test

NOTE ⓘ

In order to have runnable tests, the ZIOSpecDefault trait must be extended by an object that implements the spec method. If we extend this trait in a class, the test runner will not be able to find the tests.

Replacing

class ZioTest1 extends JUnitRunnableSpec {
  // ...
}

with

object ZioTest1 extends JUnitRunnableSpec {
  // ...
}

Should make your test start running with sbt test.

I think you copy pasted a test from the ScalaTest ones and you changed the traits mixed to the class, but forgot to make it an object. The same problem happens when you make an object a test from ScalaTest. In the previous example I provided, you can see that difference

Creating the following files, I was able to run all the tests with the command sbt test.

build.sbt

libraryDependencies ++= Seq(
  "org.scalactic"     %% "scalactic"         % "3.2.16",
  "org.scalatest"     %% "scalatest"         % "3.2.16"   % Test,
  "org.scalatestplus" %% "scalacheck-1-17"   % "3.2.16.0" % Test,
  "dev.zio"           %% "zio"               % zioVersion,
  "dev.zio"           %% "zio-test"          % zioVersion % Test,
  "dev.zio"           %% "zio-test-sbt"      % zioVersion % Test,
  "dev.zio"           %% "zio-test-magnolia" % zioVersion % Test,
  "dev.zio"           %% "zio-test-junit"    % zioVersion % Test
),
testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework")

ZioHelloWorld.scala

import zio._
import java.io.IOException

object ZioHelloWorld {
  def sayHello: ZIO[Any, IOException, Unit] =
    Console.printLine("Hello, World!")
}

ZioHelloWorldSpec.scala

import ZioHelloWorld._
import zio.test._

object ZioHelloWorldSpec extends ZIOSpecDefault {
  def spec =
    suite("ZioHelloWorldSpec")(test("sayHello correctly displays output zio test") {
      for {
        _ <- sayHello
        output <- TestConsole.output
      } yield assertTrue(output == Vector("Hello, World!\n"))
    })
}

ZioJunitHelloWorldSpec.scala

you have two ways of Integrating ZIO Test with JUnit. One is as the example added below, the other one is adding the annotation @RunWith(classOf[ZTestJUnitRunner]) above the class definition, mix ZIOSpecDefault instead of JUnitRunnableSpec and also add the library "com.github.sbt" % "junit-interface" % "0.13.3" % Test

import ZioHelloWorld._
import zio.test._
import zio.test.junit.JUnitRunnableSpec

object ZioJunitHelloWorldSpec extends JUnitRunnableSpec {
  def spec =
    suite("ZioJunitHelloWorldSpec")(test("sayHello correctly displays output using junit") {
      for {
        _ <- sayHello
        output <- TestConsole.output
      } yield assertTrue(output == Vector("Hello, World!\n"))
    })
}

HelloScalaTest.scala

import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.matchers.should.Matchers
import org.scalatestplus.scalacheck.ScalaCheckPropertyChecks

class HelloScalaTest extends AnyFunSuite with Matchers with ScalaCheckPropertyChecks {

  test("1 should be 1") {
    1 should be(1)
  }

  test("(a + b) should be (b + a)") {
    forAll { (a: Int, b: Int) =>
      (a + b) should be(b + a)
    }
  }

}

Once I executed sbt test I got the following output

+ ZioHelloWorldSpec
Hello, World!
Hello, World!
  + sayHello correctly displays output zio test
+ ZioJunitHelloWorldSpec
  + sayHello correctly displays output using junit
[info] HelloScalaTest:
[info] - 1 should be 1
[info] - (a + b) should be (b + a)
4 tests passed. 0 tests failed. 0 tests ignored.