Kotest: Execution Order of (Nested) Lifecycle Hooks

I'm confused about the order in which lifecycle hook in Kotest are executed, especially when it comes to nested tests. Consider the following example:

import io.kotest.core.spec.style.DescribeSpec

class KotestTest : DescribeSpec({
    beforeSpec { println("beforeSpec") }
    afterSpec { println("afterSpec") }
    beforeAny { println("beforeAny") }
    afterAny { println("afterAny") }
    beforeContainer { println("beforeContainer") }
    afterContainer { println("afterContainer") }
    beforeEach { println("beforeEach") }
    afterEach { println("afterEach") }
    beforeTest { println("beforeTest") }
    afterTest { println("afterTest") }
    describe("1") {
        println("1")
        beforeAny { println("beforeAny 1") }
        afterAny { println("afterAny 1") }
        beforeContainer { println("beforeContainer 1") }
        afterContainer { println("afterContainer 1") }
        beforeEach { println("beforeEach 1") }
        afterEach { println("afterEach 1") }
        beforeTest { println("beforeTest 1") }
        afterTest { println("afterTest 1") }
        describe("2") {
            println("2")
            beforeAny { println("beforeAny 2") }
            afterAny { println("afterAny 2") }
            beforeContainer { println("beforeContainer 2") }
            afterContainer { println("afterContainer 2") }
            beforeEach { println("beforeEach 2") }
            afterEach { println("afterEach 2") }
            beforeTest { println("beforeTest 2") }
            afterTest { println("afterTest 2") }
            it("3") { println("3") }
        }
    }
})

When I run this test with Kotest 5.4.1, I get the following result:

beforeSpec
beforeContainer
beforeAny
beforeTest
1
beforeContainer
beforeContainer 1
beforeAny
beforeAny 1
beforeTest
beforeTest 1
2
beforeEach
beforeEach 1
beforeEach 2
beforeAny
beforeAny 1
beforeAny 2
beforeTest
beforeTest 1
beforeTest 2
3
afterTest
afterTest 1
afterTest 2
afterAny
afterAny 1
afterAny 2
afterEach
afterEach 1
afterEach 2
afterTest
afterTest 1
afterAny
afterAny 1
afterContainer
afterContainer 1
afterTest
afterAny
afterContainer
afterSpec

Now for me the following questions arise:

Is there some documentation or guarantee about the order of beforeContainer, beforeAny and beforeTest resp. beforeEach, beforeAny and beforeTest?
I understand that the before hooks are executed from outside to inside, but would expect the after hooks to then be executed in the reverse order, i.e., from inside to outside. Is this a bug?

Motivation: We wanted to verify mock invocations like

val mock = mockk<MyClass>()

afterEach { confirmVerified(mock); clearMocks(mock); }

describe("fun1") {
    beforeEach { every { mock.fun1()} returns 1 }
    afterEach { verify { mock.fun1()} }

    it("...") {...}
}

but due to the execution order the outer afterEach gets executed first.

Solution

The order of lifecycle hooks in Kotest is confusing because one would expect that they register the hooks somewhere inside a scoped context, but instead it is registered in the spec.

You can check this by calling

println(registeredExtensions())

in any test case. The function registeredExtensions() gives you the list of all extensions that are registered in your spec, and that list will contain BeforeX and AfterX extensions in the same order, you called functions beforeX and afterX, respectively.

The problem here is that all extensions are always executed in the order they are registered. This is okay for all before-functions, but is not okay for the after-functions.

The simplest way I found to solve this, is to write a TestCaseExtension that allows to wrap something around the execution of each test:

class NestedAfter(
    private val descriptor: Descriptor,
    private val afterTest: AfterTest,
    private val type: TestType? = null,
) : TestCaseExtension {
    override suspend fun intercept(testCase: TestCase, execute: suspend (TestCase) -> TestResult): TestResult {
        return execute(testCase).also { testResult ->
            if ((type == null || type == testCase.type) && descriptor.isAncestorOf(testCase.descriptor))
                afterTest(Tuple2(testCase, testResult))
        }
    }
}

The extension executes each test and after that it conditionally calls the afterTest function. Since all extensions are called in order of their registration, each subsequent NestedAfter is wrapped inside preceding NestedAfter calls - leading to a proper nesting and the execution order we are expecting.

The condition descriptor.isAncestorOf(testCase.descriptor) makes sure that we call the afterTest function only for test cases that are (possibly nested) inside the test where we defined the extension.

This would still be quite cumbersome to use, so we also define the following functions:

fun TestScope.nestedAfterAny(afterTest: AfterTest) {
    testCase.spec.extension(NestedAfter(testCase.descriptor, afterTest))
}
fun TestScope.nestedAfterEach(afterTest: AfterTest) {
    testCase.spec.extension(NestedAfter(testCase.descriptor, afterTest, TestType.Test))
}
fun TestScope.nestedAfterContainer(afterTest: AfterTest) {
    testCase.spec.extension(NestedAfter(testCase.descriptor, afterTest, TestType.Container))
}

These function makes sure to register an instance of our extension accompanied by the descriptor of the TestCase that is currently running.

Now, if we want to have the expected execution order, then we can use nestedAfterAny instead of afterAny, nestedAfterContainer instead of afterContainer and nestedAfterEach instead of afterEach.

You can then write the following inside your test class, and the functions will respect the nesting of your test cases:

val mock = mockk<MyClass>()
describe("root") {
    nestedAfterEach { confirmVerified(mock); clearMocks(mock); }

    describe("fun1") {
        beforeEach { every { mock.fun1() } returns 1 }
        nestedAfterEach { verify { mock.fun1() } }

        it("...") {
            mock.fun1()
        }
    }
}

Note that we can call the nestedAfterX functions only inside TestContext which is why I introduced the describe("root") container in the example above.

If we want to get rid of that limitation, we can alter the NestedAfter class to have a nullable descriptor and consequently change the condition to descriptor == null || descriptor.isAncestorOf(testCase.descriptor).

Then we can define functions

fun DslDrivenSpec.nestedAfterAny(afterTest: AfterTest) {
    extension(NestedAfter(null, afterTest))
}

fun DslDrivenSpec.nestedAfterEach(afterTest: AfterTest) {
    extension(NestedAfter(null, afterTest, TestType.Test))
}

fun DslDrivenSpec.nestedAfterContainer(afterTest: AfterTest) {
    extension(NestedAfter(null, afterTest, TestType.Container))
}

which allow to define nestedAfterX functions on the root level.

Then we can alter your KotestTest from above to look like this:

class KotestTest : DescribeSpec({
    beforeSpec { println("beforeSpec") }
    afterSpec { println("afterSpec") }
    beforeAny { println("beforeAny") }
    nestedAfterAny { println("afterAny") }
    beforeContainer { println("beforeContainer") }
    nestedAfterContainer { println("afterContainer") }
    beforeEach { println("beforeEach") }
    nestedAfterEach { println("afterEach") }
    beforeTest { println("beforeTest") }
    nestedAfterEach { println("afterTest") }
    describe("1") {
        println("1")
        beforeAny { println("beforeAny 1") }
        nestedAfterAny { println("afterAny 1") }
        beforeContainer { println("beforeContainer 1") }
        nestedAfterContainer { println("afterContainer 1") }
        beforeEach { println("beforeEach 1") }
        nestedAfterEach { println("afterEach 1") }
        beforeTest { println("beforeTest 1") }
        nestedAfterEach { println("afterTest 1") }
        describe("2") {
            println("2")
            beforeAny { println("beforeAny 2") }
            nestedAfterAny { println("afterAny 2") }
            beforeContainer { println("beforeContainer 2") }
            nestedAfterContainer { println("afterContainer 2") }
            beforeEach { println("beforeEach 2") }
            nestedAfterEach { println("afterEach 2") }
            beforeTest { println("beforeTest 2") }
            nestedAfterEach { println("afterTest 2") }
            it("3") { println("3") }
        }
    }
})

The output of this test where we replaced all after functions with nestedAfter functions (except for afterSpec where it is not necessary) looks like this:

beforeSpec
beforeContainer
beforeAny
beforeTest
1
beforeContainer
beforeContainer 1
beforeAny
beforeAny 1
beforeTest
beforeTest 1
2
beforeEach
beforeEach 1
beforeEach 2
beforeAny
beforeAny 1
beforeAny 2
beforeTest
beforeTest 1
beforeTest 2
3
afterTest 2
afterEach 2
afterAny 2
afterTest 1
afterEach 1
afterAny 1
afterTest
afterEach
afterAny
afterContainer 1
afterAny 1
afterContainer
afterAny
afterContainer
afterAny
afterSpec

Also please note that afterTest is only a soft-deprecated synonym of afterEach which is why I did not provide a nested counterpart for it.