azure-devops continuous-integration azure-pipelines testcomplete azure-agent

Running TestComplete tests with VS Test / Azure Pipelines - no tests are found/run?

We're trying to set up an Azure DevOps pipeline for building and testing our codebase nightly. A portion of our test suite is TestComplete, an automated UI testing framework (https://smartbear.com/product/testcomplete/).

When we get to the TestComplete portion of the pipeline, we have a task to install the TestComplete Test Adapter and a following task that runs the TestComplete project suite (.pjs). See a snippet of those 2 tasks from our YML pipeline file:

pipeline.yml

# TestComplete - Install TestComplete Test Adapter for VSTest
- task: InstallTestCompleteAdapter@1
  displayName: Install TestComplete Test Adapter

# TestComplete - Test Project with Automated Testing
- task: VSTest@2
  displayName: TestComplete - Test Project with Automated Testing
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: '**\Project.pjs'
    searchFolder: '$(System.DefaultWorkingDirectory)\Test\Automation'
    vsTestVersion: '16.0'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    testRunTitle: 'Project Automated Tests (TestComplete)'

When these steps are performed, the logs produced by the Azure Pipeline are as follows:

Log_InstallTestCompleteAdapter.txt:

##[section]Starting: Install TestComplete Test Adapter
==============================================================================
Task         : Enable support for TestComplete tests
Description  : Installs the modules required to run TestComplete tests by using Visual Studio Test tasks. You can add a Visual Studio Test task after this task and specify a TestComplete project suite (.pjs) as a test source.
Version      : 1.0.2180
Author       : SmartBear Software
Help         : 
==============================================================================
##[section]Finishing: Install TestComplete Test Adapter

Log_TestCompleteVSTest2.txt:

##[section]Starting: TestComplete - Test Project with Automated Testing
==============================================================================
Task         : Visual Studio Test
Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
Version      : 2.153.9
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
==============================================================================
SystemVssConnection exists true
##[warning]Unable to get feature flag: TestExecution.EnableHydra
SystemVssConnection exists true
##[warning]Unable to get feature flag: TestExecution.EnableTranslationApi
SystemVssConnection exists true
##[warning]Unable to get feature flag: TestExecution.EnableDiagnostics
Running tests using vstest.console.exe runner.
======================================================
Test selector : Test assemblies
Test filter criteria : null
Search folder : C:\agent\_work\1\s\Test\Automation
VisualStudio version selected for test execution : 16.0
Attempting to find vstest.console from a visual studio installation.
Run in parallel : false
Run in isolation : false
Path to custom adapters : null
Other console options : null
Code coverage enabled : false
Diagnostics enabled : false
SystemVssConnection exists true
Run the tests locally using vstest.console.exe
========================================================
Test selector : Test assemblies
Test assemblies : **\Project.pjs
Test filter criteria : null
Search folder : C:\agent\_work\1\s\Test\Automation
Run settings file : C:\agent\_work\1\s
Run in parallel : false
Run in isolation : false
Path to custom adapters : null
Other console options : null
Code coverage enabled : false
Diagnostics enabled : false
Rerun failed tests: false
VisualStudio version selected for test execution : 16.0
Attempting to find vstest.console from a visual studio installation.
========================================================
======================================================
[command]"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" @C:\agent\_work\_temp\014ff0d1-411c-11ee-9460-2b0611124091.txt
Microsoft (R) Test Execution Command Line Tool Version 16.9.1
Copyright (c) Microsoft Corporation.  All rights reserved.

vstest.console.exe 
"C:\agent\_work\1\s\Test\Automation\Project\Project.pjs"
/logger:"trx"
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: C:\agent\_work\1\s\TestResults\admin_machine_2023-08-22_13_45_03.trx

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.
No Result Found to Publish 'C:\agent\_work\1\s\TestResults\admin_machine_2023-08-22_13_45_03.trx'.
##[section]Async Command Start: Publish test results
##[section]Async Command End: Publish test results
##[section]Finishing: TestComplete - Test Project with Automated Testing

As you can see, the vstest.console.exe process was able to find the .pjs project file of interest at "C:\agent_work\1\s\Test\Automation\Project\Project.pjs", but after that no tests were run/performed. So I added a step to copy out the contents of the "C:\agent_work\1\s\TestResults" directory to a directory outside of the working directory ("C:\TestResults") to observe the produced .trx file, and this is the result of that:

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="a5e4dd47-6b28-454c-86d1-c16554535717" name="machine@admin 2023-08-22 16:37:47" runUser="machine\admin" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2023-08-22T16:37:47.3236283-05:00" queuing="2023-08-22T16:37:47.3236283-05:00" start="2023-08-22T16:37:47.2425064-05:00" finish="2023-08-22T16:37:47.3236283-05:00" />
  <TestSettings name="default" id="3ec85379-7cf6-49a0-8f01-01b0c9ed2056">
    <Deployment runDeploymentRoot="admin_machine_2023-08-22_16_37_47" />
  </TestSettings>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="0" executed="0" passed="0" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
  </ResultSummary>
</TestRun>

So it looks like although the .pjs file is getting found, no elements are getting added to any of the elements (I'm assuming these have child nodes in a valid test run).

I've tried to follow some of SmartBear's documentation to determine how/why no tests are being identified in the .pjs file that is being found:

Ensuring TestComplete can be launched from my agent manually (e.g. it's licensed and can be opened)
Specifying tests I want to run by adding the // @"test name" attribute above the corresponding javascript functions
Adding tests to my execution plan and ensuring the "Test Case" checkbox is checked.
Ensuring my agent is configured properly (autologon=true, interactive=true)

SmartBear documentation sources:

Any ideas why the vstest.console.exe process isn't queuing up any tests?

Solution

That's just an idea that might help get around the problem:

We set up a pipe line to run automated tests with TestComplete using an old fashioned 'batch file' approach. I mean, we really call TestComplete via command line in the pipeline from a script step. In the mentioned batch file, we tell TestComplete to save the logs to $(Build.SourcesDirectory)\Logs. Later on, we publish those logs to see results and identify possible issues. (See https://support.smartbear.com/testcomplete/docs/working-with/automating/command-line-and-exit-codes/command-line.html for details on command line for TestComplete.)

We have an image, where TestComplete and other things like GIT etc. are already installed and licensed. We did in our yaml file the following:

Preparation stage: Here we check out (pull) the sources, in particular the test suite, using GIT command line and prepare other things our application under test need to run properly.
Running Tests stage: We just call a runTests.bat file in a script step. The runTests.bat is located directly in the $(Build.SourcesDirectory) and write logs to the $(Build.SourcesDirectory)\Logs directory. In the runTests.bat itself we build a appropiated command line for run our suite and the regarded tests as needed. (In our case, using tags, which may be irrelevant for you.)
To save the test results, wie zip the test log using a task ArchiveFiles@2 to create an artifact and then, we publish the artifact using a task PublishPipelineArtifact@1 to be able to access the result after pipele runs.

That's the idea.

Hints on the command line we build for TestComplete:

We always gave the follwoing parameters: /run /Exit /SilentMode /DoNotShowLog /exit /SelfHealing:Off
We set explicitly the Log files using the /ExportLog and /ErrorLog switches, where the files are both in the $(Build.SourcesDirectory)\Logs directory.

Hint on publishing logs: This is a copy and paste of the last two steps in the pipeline to save the test logs:

      - task: ArchiveFiles@2
        inputs:
          rootFolderOrFile: '$(Build.SourcesDirectory)\Logs'
          includeRootFolder: false
          archiveType: 'zip'
          archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
          replaceExistingArchive: true
        displayName: 'Create TestComplete Result Artifact'
        condition: always()
        name: 'archive'
      - task: PublishPipelineArtifact@1
        inputs:
          targetPath: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
          artifactName: 'testCompleteResult'
          artifactType: 'pipeline'
          publishLocation: 'Container'
        displayName: 'Publish TestComplete Result Artifact'
        condition: always()
        name: 'publish'

I hope this helps.