junitbamboo

Why is the bamboo junit parser rendering "unnamed test suite"

I'm seeing several test cases showing up in a bamboo JUnit Parser step with no test suite name being rendered for the test cases. What can explain this behavior?

multiple test cases with unnamed test suite

Solution

  • Playing around with a dummy test result we can see that bamboo has at least two forms of test suite naming detection.

    Explicitly named Testsuite

    The most sensible parsing operation happens under an explicitly named test suite. In the xml this shows by the name attribute in the testsuite tag.

    <?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="test_dummy_suite_name" tests="1" failures="0" errors="0">
    <testcase name="test_dummy_case_name" status="run" duration="0.001" time="1"></testcase>
  </testsuite>
</testsuites>

    In this circumstance bamboo properly parses the testsuite's name as seen here: explicit named testsuite

    Pytest generated xml

    Pytest when it generates junit xml, via the --junit-xml=xml_path.xml argument, has a convention of injecting the testsuite name with the generic pytest string when left to the default value for its junit_suite_name.

    <?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite errors="0" failures="1" hostname="XXX" name="pytest" skipped="0" tests="3" time="0.038" timestamp="2022-03-03T17:51:33.038037">
        <testcase classname="classnameX.classnameY" file="junit_explore/test_module.py" line="3" name="test_passing1" time="0.001"></testcase>
        <testcase classname="junit_explore.test_module" file="junit_explore/test_module.py" line="6" name="test_passing2" time="0.000"></testcase>
        <testcase classname="" file="junit_explore/test_module.py" line="6" name="test_passing_empty_classname" time="0.000"></testcase>
    </testsuite>
</testsuites>

    Bamboo appears to be familiar with this convention and will actually fallback to parsing the classname attribute for the testcases to tokenize on the . character to extract the substring which follows. Note the following output from the above xml:

    example of test suite name parsed from classname attribute

    We can see that for the test cases with an empty classname attribute Bamboo robustly handles that case but ultimately cannot determine the test suite name and falls back to the unnamed test suite representation since that's all the context it has for such test cases.

    backstory: it turns out that running pytest junit generation from a bazel execution somehow strips or interferes with the classname generation. It's not entirely clear why this is the case to me at this point in time. pytest generates the value for this attribute in the following source https://github.com/pytest-dev/pytest/blob/55debfad1f690d11da3b33022d55c49060460e44/src/_pytest/junitxml.py#L126. I may be able to trace up through the codebase to see if anything can be determined there.

    Backstory update 3/21/2022 I wound up digging into the bazel behavior and authoring an instrumented build of nodes.py and essentially found the session root dir could not be established with their implementation of relative path logic session.config.rootdir. See https://github.com/pytest-dev/pytest/discussions/9807 for details.