unit-testingazure-devopsnunitvnext

How to include NUnit test output when publishing test results in Azure DevOps?

I have a vNext build which runs a few NUnit tests and publishes the results. However, the output produced by the tests is not published, even though it is found in the respective XML file.

For example:

enter image description here

But, if I inspect the respective XML files - it is there:

enter image description here

Notice the <output> element.

I would very much like to see this output published along the test results. How can I do it?

P.S.

There is another problem with the publishing. The Run Duration is reported as 4h 59m in the top pane, but in the bottom details pane it is the correct 10:48.176 minutes. The 4h 59m looks very much as 5h, which is the time difference between EST and UTC. The tests were run on an Octopus server and were fetched by the vNext build. Maybe there is a time zone confusion somewhere.

EDIT 1

We have an on-premises TFS

EDIT 2

Examining the source code of the Publish Test Results task reveals that it uses Microsoft.TeamFoundation.TestClient.PublishTestResults assembly to parse the NUnit XML test result. Specifically the following C# code is used to parse the test-case element (NUnit3ResultsXmlReader.cs):

if (testCaseResultNode.Attributes["result"] != null)
{
  testCaseResultData.TestCaseResult.Outcome = !string.Equals(testCaseResultNode.Attributes["result"].Value, "Passed", StringComparison.OrdinalIgnoreCase) ? (!string.Equals(testCaseResultNode.Attributes["result"].Value, "Failed", StringComparison.OrdinalIgnoreCase) ? (!string.Equals(testCaseResultNode.Attributes["result"].Value, "Skipped", StringComparison.OrdinalIgnoreCase) ? TestOutcome.Inconclusive.ToString() : TestOutcome.NotExecuted.ToString()) : TestOutcome.Failed.ToString()) : TestOutcome.Passed.ToString();
  XmlNode xmlNode1 = testCaseResultNode.SelectSingleNode("failure");
  if (xmlNode1 != null)
  {
    XmlNode xmlNode2 = xmlNode1.SelectSingleNode("message");
    XmlNode xmlNode3 = xmlNode1.SelectSingleNode("stack-trace");
    testCaseResultData.TestCaseResult.ErrorMessage = xmlNode2 != null ? xmlNode2.InnerText : (string) null;
    testCaseResultData.TestCaseResult.StackTrace = xmlNode3 != null ? xmlNode3.InnerText : (string) null;
    XmlNode xmlNode4 = testCaseResultNode.SelectSingleNode("output");
    if (!string.IsNullOrWhiteSpace(xmlNode4 != null ? xmlNode4.InnerText : (string) null))
      testCaseResultData.ConsoleLog = xmlNode4.InnerText;
  }
}

From which it follows that the assembly authors think that test output is only useful when the test-case in question has failed. This is an unfortunate decision, because it is not up to them to decide when the output is useful. If it is in the test results XML, then it should be published.

Opened a new issue https://github.com/Microsoft/azure-pipelines-tasks/issues/8979

Opened an Azure DevOps feature request - https://developercommunity.visualstudio.com/idea/432166/the-publish-tests-azure-devops-plugin-should-publi.html

Solution

  • My solution to this unfortunate behavior force upon us is to mangle the XML test result before handing it off to the Publish Test Results task.

    The following powershell code does the trick:

    $Modified = $false
$xml = [xml](cat $OriginalTestResultsXmlFile -Raw)
$xml.SelectNodes('//test-case') |? { !$_.failure -and $_.output } |% { 
    $Modified = $true
    $_.InnerXml = @"
<!-- Workaround the issue https://github.com/Microsoft/azure-pipelines-tasks/issues/8979. No real failure here -->
<failure/>
$($_.InnerXml)

"@
}

if ($Modified)
{
    $xml.Save($TestResultsXmlFile)
}
else
{
    move $OriginalTestResultsXmlFile $TestResultsXmlFile
    $OriginalTestResultsXmlFile = $TestResultsXmlFile
}

    We have to trick the task into executing the logic we need by creating a dummy failure element besides the output element.

    Lo and behold:

    enter image description here