I have a Spock @Unroll
test like this:
@Unroll
def 'A session uses the TLS protocol #enabledProtocolsOfSocket #protocolOfSession'() {
given:
// ...
expect:
// ...
where:
enabledProtocolsOfSocket | protocolOfSession
["TLSv1.3"] | "TLSv1.3"
["TLSv1.2", "TLSv1.3"] | "TLSv1.3"
["TLSv1.2"] | "TLSv1.2"
["TLSv1.2", "TLSv1.1"] | "TLSv1.2"
["TLSv1.1", "TLSv1.2"] | "TLSv1.2"
["TLSv1.1"] | "NONE"
["TLSv1"] | "NONE"
["TLSv1", "TLSv1.1"] | "NONE"
["TLSv1.1", "TLSv1"] | "NONE"
}
which leads to this JUnit result XML:
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="com.acme.TlsProtocolSpec" tests="9" skipped="0" failures="0" errors="0" timestamp="2024-07-19T17:23:47" hostname="acme-build5eb650" time="0.403">
<properties/>
<testcase name="A session uses the TLS protocol [TLSv1.3] TLSv1.3" classname="com.acme.TlsProtocolSpec" time="0.081"/>
<testcase name="A session uses the TLS protocol [TLSv1.2, TLSv1.3] TLSv1.3" classname="com.acme.TlsProtocolSpec" time="0.037"/>
<testcase name="A session uses the TLS protocol [TLSv1.2] TLSv1.2" classname="com.acme.TlsProtocolSpec" time="0.084"/>
<testcase name="A session uses the TLS protocol [TLSv1.2, TLSv1.1] TLSv1.2" classname="com.acme.TlsProtocolSpec" time="0.07"/>
<testcase name="A session uses the TLS protocol [TLSv1.1, TLSv1.2] TLSv1.2" classname="com.acme.TlsProtocolSpec" time="0.071"/>
<testcase name="A session uses the TLS protocol [TLSv1.1] NONE" classname="com.acme.TlsProtocolSpec" time="0.014"/>
<testcase name="A session uses the TLS protocol [TLSv1] NONE" classname="com.acme.TlsProtocolSpec" time="0.013"/>
<testcase name="A session uses the TLS protocol [TLSv1, TLSv1.1] NONE" classname="com.acme.TlsProtocolSpec" time="0.013"/>
<testcase name="A session uses the TLS protocol [TLSv1.1, TLSv1] NONE" classname="com.acme.TlsProtocolSpec" time="0.013"/>
<system-out><![CDATA[]]></system-out>
<system-err><![CDATA[]]></system-err>
</testsuite>
The problem is: I cannot map these results to the test in post-processing the result file – there is no result named A session uses the TLS protocol #enabledProtocolsOfSocket #protocolOfSession
(due to the placeholders) and there are no tests named A session uses the TLS protocol [TLSv1.2, TLSv1.3] TLSv1.3
or ... or ... (due to the placeholders being replaced).
Is there some feature to "group" or "merge" these test runs into one result, named as the test? I'm thinking of something like xUnit's <collection>
Element, which contains one <test>
child node for each test run, like so:
<collection total="9" passed="9" failed="0" skipped="0" name="Test collection for Com.Acme.SomeClass.Tests.SomeTestName" time="0.484">
<test name="Com.Acme.SomeClass.Tests.SomeTestName(someParam: "10")" ...>
</test>
<test name="Com.Acme.SomeClass.Tests.SomeTestName(someParam: "11")" ...>
</test>
...
</collection>
But any solution to preserve the relation of the generically named test to all test runs (i.e. if just one test run fails, the test is reported failing) is welcome, be it by configuring the JUnit output or by some Spock feature.
As @leonard-brünings pointed out: The build tool is of interest here, as it produces said output. I'm currently using Gradle 8.5 (Groovy 3.0.17, Spock 2.4-M4-groovy-4.0), configured like this:
test {
reports {
junitXml.required = true
html.required = true
}
useJUnitPlatform()
}
I searched a bit in that direction and found settings like mergeReruns
(which sounds helpful but is for failed and then retried runs?) and plugins like The Test Report Aggregation Plugin (which aggregates by type, not by testcase?) here. So at least these do not help.
Seems that @Unroll
is obsolete since Spock 2.x:
A simple
@Unroll
without argument is not needed anymore [...], so any simple@Unroll
annotations can be removed from existing code.
But/and: There's @Rollup
now, which reports a single <testcase>
now, which is what I was looking for.
On the downside: Those rolled up tests do not show up individually in the IDE neither; all those #myParam1
-placeholders in the test's name are obsolete now, as well (I guess).