How to place Azure pipeline run to the end of the current queue?

I have a huge project with UI tests, and we want to automate the test runs via Azure DevOps. There are about 20 huge E2E's each takes from 20 minutes to 5 hours to run. And each release all these tests should be passed. The problem is the timeout threshold for pipelines is just 1 hour, and in our project we cannot change it, so we have overcame this problem by splitting some big E2E's in parts. As example there are 5 pipelines for one test and all these parts should be completed from 1st to 5th in sequence. 2nd cannot be started before 1st is finished.

So now we have about 50 pipelines for all tests we need to run each release. Manually its painful.

We have assigned to all these pipelines only 1 agent, that cannot run parallel jobs, so all runs would be placed in queue, but this queue can be mixed up sometimes.

Problem: If I run 10 pipelines in a row instantly, they would be completed in the same sequence I pressed Run (or sent post api: run build) for each pipeline. But If i want to add some pipelines when >2 minutes passed after i ran 1st sequence, these pipelines will mix up with previous queue this way:

I cannot run another test before all parts of previous one completed (this will change the data on mobile test device and will lead to test fail).

Is there any way to add pipeline runs to the end of current queue?

If I run 10 pipelines in a row instantly, they would be completed in the same sequence I pressed Run (or sent post api: run build) for each pipeline. But If I want to add some pipelines when >2 minutes passed after i ran 1st sequence, these pipelines will mix up with previous queue

A pipeline can have dependencies between stages and/or jobs. This means that not all jobs can be added to the queue immediately, i.e. they need to wait for its dependencies to finish. If in the meantime another pipeline starts, all jobs that are ready to run will be added to the queue - that's why some jobs from the second pipeline might start before all jobs from the first pipeline finish.

Considering that, your best option is probably to wait for each pipeline to finish before starting the next one in the sequence.

Consider:

• Using Pipeline triggers to trigger one pipeline upon the completion of another.
• Queuing the next pipeline in the sequence programmatically

Using pipeline triggers

# app-ci YAML pipeline
# We are setting up a pipeline resource that references the security-lib-ci
# pipeline and setting up a pipeline completion trigger so that our app-ci
# pipeline runs when a run of the security-lib-ci pipeline completes
resources:
  pipelines:
  - pipeline: securitylib # Name of the pipeline resource.
    source: security-lib-ci # The name of the pipeline referenced by this pipeline resource.
    project: FabrikamProject # Required only if the source pipeline is in another project
    trigger: true # Run app-ci pipeline when any run of security-lib-ci completes

steps:
  # ...

Queuing a pipeline programmatically

You can queue the next pipeline in the sequence programmatically as the last step of the current pipeline.

Using az pipelines run command of the Azure DevOps CLI:

- script: >-
    az pipelines run
    --name 'my-pipeline'
    --org $(System.CollectionUri)
    --project $(System.TeamProject)
    --branch $(Build.SourceBranch)
  displayName: 'Run pipeline'
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

Using the Trigger Build task:

- task: TriggerBuild@3
  displayName: 'Trigger a new build'
  inputs:
    buildDefinition: 'Your build name'
    useSameBranch: false
    branchToUse: master
    waitForQueuedBuildsToFinish: true
    authenticationMethod: 'OAuth Token'
    password: $(System.AccessToken)