azurepowershellunit-testingcode-coveragecoverlet

coverlet does not create report in azure devops

I am using

and dotnet sdk v6.0.402

When I run this test command in powershell (restore and build ran before that)

dotnet test --no-build --no-restore --collect:"XPlat Code Coverage" /p:Configuration=$Cfg /p:CollectCoverage=true /p:CoverletOutput=.\CodeCoverage\ --% /p:CoverletOutputFormat=\"cobertura,opencover\"

The report files

are created and the powershell output is this:

Test run for C:\Users\MyUser\repos\My.Project\Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Attachments:
  C:\Users\MyUser\repos\My.Project\Specs\TestResults\dbe8cb53-3ec5-4227-a231-3bfedf94694f\coverage.cobertura.xml
Passed!  - Failed:     0, Passed:    14, Skipped:     0, Total:    14, Duration: 910 ms - My.Project.Specs.dll (net472)

Calculating coverage result...
  Generating report '.\CodeCoverage\coverage.cobertura.xml'
  Generating report '.\CodeCoverage\coverage.opencover.xml'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
| My.Project           | 20.23% | 18.53% | 20.09% |
+----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 20.23% | 18.53% | 20.09% |
+---------+--------+--------+--------+
| Average | 20.23% | 18.53% | 20.09% |
+---------+--------+--------+--------+

I have this azure-pipeline task:

- task: DotNetCoreCLI@2
  displayName: Test
  inputs:
    command: test
    arguments: '--no-restore --no-build --collect:"XPlat Code Coverage" /p:Configuration=$(Build.Configuration) /p:CollectCoverage=true  /p:CoverletOutput=$(Build.SourcesDirectory)\CodeCoverage --% /p:CoverletOutputFormat=\"cobertura,opencover\"'
  publishTestResults: true

Which executes this command:

C:\agent\_work\_tool\dotnet\dotnet.exe test --logger trx --results-directory C:\agent\_work\_temp --no-restore --no-build "--collect:XPlat Code Coverage" /p:Configuration=Release /p:CollectCoverage=true /p:CoverletOutput=C:\agent\_work\9\s\CodeCoverage --% "/p:CoverletOutputFormat=\cobertura,opencover\""

And has this output

Test run for C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\My.Project.Specs.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\LivingDoc.SpecFlowPlugin.dll
-> Loading plugin C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TechTalk.SpecFlow.xUnit.SpecFlowPlugin.dll
-> Loading plugin C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\a4203b08-db42-449b-86d7-55cb48c54fc4\a4203b08-db42-449b-86d7-55cb48c54fc4\assembly\dl3\e501d05b\fee426f7_ddf4d801\My.Project.Specs.dll
-> Using specflow.json
-> LivingDocPlugin: Output generated in: C:\agent\_work\9\s\My.Project.Specs\bin\Release\net472\TestExecution.json
Results File: C:\agent\_work\_temp\BUILDMACHINE01$_BUILDMACHINE01_2022-11-10_09_25_37.trx

Passed!  - Failed:     0, Passed:    14, Skipped:     0, Total:    14, Duration: 985 ms - My.Project.Specs.dll (net472)

Attachments:
  C:\agent\_work\_temp\faa5dbb6-5931-43fe-880e-a37576815c1c\coverage.cobertura.xml
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
Async Command Start: Publish test results
Publishing test results to test run '1436028'.
TestResults To Publish 12, Test run id:1436028
Test results publishing 12, remaining: 0. Test run id: 1436028
Published Test Run : https://dev.azure.com/orgteamservices/My.Project/_TestManagement/Runs?runId=1436028&_a=runCharts
Async Command End: Publish test results
Finishing: Test

No reports are generated on the on-prem agent machine in the expected directory. The agent does create this though C:\agent\_work\_temp\faa5dbb6-5931-43fe-880e-a37576815c1c\coverage.cobertura.xml

Why is it not creating both reports in the expected directory?

Solution

  • The problem:

    Since Visual Studio Build tools 17.4.0 any arguments (/p:) passed to coverlet are ignored.

    The workaround:

    Specifying publishTestResults: true as input parameter for DotNetCoreCLI@2 task (or no publishTestResults input at all) causes the following arguments to be added: --logger trx --results-directory $(Agent.TempDirectory). This is normal behaviour btw.

    This in turn causes the coverage results to be placed in the agents temp directory as the coverlet arguments are ignored.

    So set publishTestResults to false

    Add the arguments which are now omitted yourself: --logger trx --results-directory CodeCoverage

    Pass any additional runsettings either by file or by adding the following runsettings arguments: --DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover,cobertura

    This should mitigate the problem for now.

    This is what my entire yml task looks like:

    - task: DotNetCoreCLI@2
      displayName: 'Run acceptance tests'
      inputs:
        command: 'test'
        projects: './src/Service.Requirements/Service.Requirements.csproj'
        workingDirectory: './src/Service.Requirements'
        publishTestResults: false
        arguments: '--logger trx --results-directory CodeCoverage --configuration $(buildConfiguration) --no-build --collect "XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Exclude=[Service.Data]*'