continuous-integrationautomated-teststeamcityspecflowspecrun

How to fix failed tests count in teamcity for re-executed tests?


We are using TeamCity for Selenium tests execution with Specflow+Specrun, the problem is that TeamCity count re-executed tests.

For example if some test failed for first time he will be re-executed two times more, in teamcity we will see that three tests failed, but probably it was one test.

Also if first re-execution will fail, but other two will success, this will be reported in teamcity as two failed, one passed, but I need to be reported that only one test has been passed.

It is possible to configure this in TeamCity using service messages or something else?

UPDATED:

Based on answer we can gather logs using powershell script and change build status using teamcity service messages:

$path = Get-ChildItem %teamcity.build.checkoutDir%\ProjectFolder\bin\Remote\TestResults\specrun.log
$file = Get-content $path
$total = $file | select-string "Total:"
$passed = $file | select-string "Succeeded:"
$failed = $file | select-string "Failed:"

write-host $( "##teamcity[buildStatus text='Tests {0}, {1}, {2}']" -f $total, $passed, $failed )

Solution

  • TeamCity is behaving as expected. TeamCity reports what Testing Framework tells him. Roughly the process is:

    You have to configure your Testing Framework (specflow+specrun in your case) to change the way it reports the re-execution. You should configure specflow to log the re-executions in a different way... if it can.

    In general you should avoid re-execution of tests. The tests should work at the first attempt. They should prepare the state if needed, do their test, and clean the state.

    The need of re-execution indicates probably some dependency with the other tests. Something like:

    The correct way is that TEST001 does not have a dependency of TEST002, adds the user Foo1, test search and clean user Foo1.