How to trigger a middle stage in yaml pipeline in Azure Devops after another pipeline's completion?

I need to trigger a yaml stage of the pipeline after completion of another pipeline completion.

I have a setup like this: the main pipeline has

stage1 -> stage2 -> stage3 -> stage4

After stage1 completes, it should trigger the testing yaml pipeline which is standalone and it has to get completed and once test yaml pipeline is completed successfully.

The next stages from stage2 should continue

If I understand it correctly, you're looking to have a single pipeline act as an orchestration that kicks off other pipelines and waits for them to complete before proceeding to the next stage. There are a few options available.

For starters, let's assume that stage1 triggers your pipeline using something like the Trigger Build extension or Azure DevOps REST API to queue the build. As we'll need to know the pipeline run id later, you'll need to create an output variable we can reference later:

- stage: stage1
  jobs:
  - job: job1
    steps:
    - ...
    - pwsh: |
        $buildId = "<value obtained from rest api or trigger build>"
        write-host "##vso[task.setvariable variable=buildId;isoutput=true]$buildId"
      name: captureId

Let's also assume that stage2 has a deployment job:

- stage: stage2
  dependsOn: stage1
  jobs:
  - deployment: deploy1
    environment: stage2_environment
    strategy:
      runOnce:
        deploy:
          steps:
          - ... deploy steps

Approvals + Checks: Invoke REST API (Synchronous)

You can use the "Approvals + Checks" feature to add a Invoke REST API check that polls the status of the build using the Azure DevOps REST API.

To set this up:

1. Create a generic service connection that contains the base URL of your Azure DevOps organization (https://dev.azure.com/<orgname>). The credentials defined here won't be used.

2. On the Pipeline Environment for stage2, add a Invoke REST-API check. Specify the following:

   • Service Connection:
   • Headers: { "Authorization": "Bearer $(System.AccessToken)" }
   • Method: GET
   • Url Suffix and parameters: $(System.TeamProject)/_apis/build/builds/$(stage1.job1.captureId.buildId)?api-version=7.1
   • Completion event: ApiResponse
   • Success criteria: eq(root['result'],'Succeeded')
   • Timeout (minutes): 60
   • Time between evaluations (minutes): 6

The above calls an Azure DevOps REST endpoint to get the details of the build. It evaluates the result property of the build. Note the gate supports using variables from previous stages.

An important limitation with this approach is the ratio between Timeout and Time between evaluations must be 10:1. Anything more than this and it will fail due to throttling measures. For longer running pipelines, this will translate into longer evaluations, if the pipeline being monitored takes up to 240 minutes, the gate will check every 24 minutes. In my opinion, this is a long time to wait for a pipeline, but your mileage may vary.

Approvals + Checks (Asynchronous)

If you need better control of the retry interval, you can use either Invoke REST API or Invoke Function App check using the asynchronous callback model to call a custom Function App that monitors your pipeline. The asynchronous pattern is the recommended approach for custom gates. Your function app could in theory check the status of the pipeline as often as you like (assuming you don't exceed the API throttling limits ...roughly 200 times in 5 minutes).

If you'd consider using a Function App to monitor your pipeline, a cleaner approach would use the Function App to queue and monitor the pipeline as this would provide more re-use options for other pipelines.