Parallel execution of tests on TFS

We use TFS on our project. I have set Parallelism -> Multi Agent in the phase settings. The command itself to run (.NET Core) is:

dotnet test --filter TestCategory="Mobile" --logger trx -m:1. 

Do I understand correctly that these settings will not split the tests between the two agents, but run the command above on the two agents?

The Visual Studio Test (- task: VSTest@2) has built-in magic to distribute the test based on configurable criteria:

You could switch to using the vstest task instead; to run your tests to get this "magic".

The dotnet core task or invoking dotnet straight from the command line doesn't have this magic.

There is a github repo that shows how to take advantage of the default of the hidden variables that are set by the agent when running in parallel:

#!/bin/bash

filterProperty="Name"

tests=$1
testCount=${#tests[@]}
totalAgents=$SYSTEM_TOTALJOBSINPHASE
agentNumber=$SYSTEM_JOBPOSITIONINPHASE

if [ $totalAgents -eq 0 ]; then totalAgents=1; fi
if [ -z "$agentNumber" ]; then agentNumber=1; fi

echo "Total agents: $totalAgents"
echo "Agent number: $agentNumber"
echo "Total tests: $testCount"

echo "Target tests:"
for ((i=$agentNumber; i <= $testCount;i=$((i+$totalAgents)))); do
targetTestName=${tests[$i -1]}
echo "$targetTestName"
filter+="|${filterProperty}=${targetTestName}"
done
filter=${filter#"|"}

echo "##vso[task.setvariable variable=targetTestsFilter]$filter"

This way you can slice the tasks in your pipeline:

  - bash: |
      tests=($(dotnet test . --no-build --list-tests | grep Test_))
      . 'create_slicing_filter_condition.sh' $tests
    displayName: 'Create slicing filter condition'
  
  - bash: |
      echo "Slicing filter condition: $(targetTestsFilter)"
    displayName: 'Echo slicing filter condition'
  - task: DotNetCoreCLI@2
    displayName: Test
    inputs:
      command: test
      projects: '**/*Tests/*Tests.csproj'
      arguments: '--no-build --filter "$(targetTestsFilter)"'

I'm not sure whether this will support 100.000's of tests. In that case you may have to break the list into batches and call dotnet test multiple times in a row. I couldn't find support for vstest playlists.