azure-devops yaml azure-pipelines cicd azure-pipelines-yaml

My Azure pipeline is not being triggered despite meeting conditions

My YAML file in my repos has enabled automated deployment for my Dev and QA environments. But for Prod, it doesn't seem to be working despite meeting the conditions.

name: $(BuildDefinitionName)-$(SourceBranchName)-$(Year:yyyy).$(Month).$(DayOfMonth)$(Rev:.r)

trigger:
  branches:
    include:
    - main
    - release/*
  paths:
    include:
      - workflows/*
      - ProductXETL/*

parameters:
- name: deployCluster
  type: boolean
  default: false

variables:
  tag: '$(Build.BuildId)'
  workingDirectory: '$(System.DefaultWorkingDirectory)'

pool:
  name: Company Analytics SelfHosted
  demands:
    - agent.name -equals px-dataagent-de  

stages:
- stage: Build
  displayName: "Build"
  jobs:
  - job: Build
    displayName: Build
    pool:
      name: Company Analytics SelfHosted
      demands:
        - agent.name -equals px-dataagent-de  
    steps:
      - task: CopyFiles@2
        displayName: Copy ARM templates to staging
        inputs:
          SourceFolder: deploy
          TargetFolder: $(Build.ArtifactStagingDirectory)
      - task: CopyFiles@2
        displayName: Copy workflows to staging
        inputs:
          SourceFolder: workflows
          TargetFolder: $(Build.ArtifactStagingDirectory)/workflows
      - task: CopyFiles@2
        displayName: Copy pyjaws to staging
        inputs:
          SourceFolder: pyjaws
          TargetFolder: $(Build.ArtifactStagingDirectory)/pyjaws
      - task: PublishPipelineArtifact@1
        displayName: 'Publish Pipeline Artifact: drop'
        inputs:
          targetPath: '$(Build.ArtifactStagingDirectory)'
          artifact: deploy

- stage: Dev
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  dependsOn: ["Build"]
  displayName: "Deploy to Dev"
  variables:
  - group: PX DataPlatform Release (Dev)
  jobs:
    - deployment:
      pool:
        name: Company Analytics SelfHosted
        demands:
          - agent.name -equals px-dataagent-de  
      displayName: "Deploy to Dev"
      environment: Dev
      strategy:
        runOnce:
          deploy:
            steps:
              - template: azure-pipeline.template.yml
                parameters:
                  serviceConnection: 'Company Analytics Dev UK'
                  subscriptionId: 'xxxxxxxxxxxxxxxxxxxxxxxxxxx'
                  deployCluster: ${{ parameters.deployCluster }}
                  environment_name: DEV
- stage: QA
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  dependsOn: ["Dev"]
  displayName: "Deploy to QA"
  variables:
  - group: PX DataPlatform Release (QA)
  jobs:
    - deployment:
      pool:
        name: Company Analytics SelfHosted
        demands:
          - agent.name -equals px-dataagent-de  
      displayName: "Deploy to QA"
      environment: QA
      strategy:
        runOnce:
          deploy:
            steps:
              - template: azure-pipeline.template.yml
                parameters:
                  serviceConnection: 'Company Analytics Dev UK'
                  subscriptionId: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx'
                  deployCluster: ${{ parameters.deployCluster }}
                  environment_name: QA

- stage: Prod
  condition: and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/heads/release/'))
  dependsOn: ["Build"]
  displayName: "Deploy to Prod"
  variables:
  - group: PF DataPlatform Release (Prod)
  jobs:
    - deployment:
      pool:
        name: Company Analytics SelfHosted Prod
        demands:
          - agent.name -equals px-dataplat-vm2  
      displayName: "Deploy to Prod"
      environment: UK Production
      strategy:
        runOnce:
          deploy:
            steps:
              - template: azure-pipeline.template.yml
                parameters:
                  serviceConnection: 'Company Analytics Prod UK'
                  subscriptionId: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
                  deployCluster: ${{ parameters.deployCluster }}
                  environment_name: PROD

In theory when I create a new branch starting with "release/", I expect it to begin building and I should be able to take it from there, but nothing is being triggered at all which contrasts my dev deployment. My dev deployment works well- as soon as I complete a PR into my main branch, deployment kicks off there and it works fine.

I thought maybe to add the second agent here:

pool:
  name: Company Analytics SelfHosted
  demands:
    - agent.name -equals px-dataagent-de

as my Dev/QA both share the same agent and Prod uses a different agent, but even after listing it below my dev agent, no luck either. I also tried using

-or:

and listed both agents below, no luck either.

I have anonymised parts of my code where appropriate, I hope it makes sense.

If someone can help me understand why my prod deployment isn't being automatically triggered I would be grateful.

Thanks.

Solution

TL;DR your build will be triggered if and only if there are changes in any of the folders specified in the path section, when using the main or release/* branches.

More details

I ran some tests using a build with a similar trigger containing paths:

trigger:
  branches:
    include:
    - main
    - release/*
  paths:
    include:
      - workflows/*
      - ProductXETL/*

In theory when I create a new branch starting with release/, I expect it to begin building

Build will not be triggered immediately when you create the branch because there are no changes in the workflows/ or ProductXETL/ folders. If you do change the files in any of these folders AFTER creating the branch the build will be triggered.

My dev deployment works well- as soon as I complete a PR into my main branch, deployment kicks off there and it works fine.

Build will only be triggered if your pull request contains changes in the workflows/ or ProductXETL/ folders.

Workaround

If the path makes sense for Dev and QA environments but not for Prod I'd suggest you create 2 separate pipelines, each one with a different trigger.

Using a base pipeline (via extends templates) will solve the problem of code duplication.

Pipeline for Dev and QA environments:

parameters:
- name: deployCluster
  type: boolean
  default: false

trigger:
  branches:
    include:
    - main
  paths:
    include:
      - workflows/*
      - ProductXETL/*

extends:
  template: base-pipeline.yaml
  parameters:
    isProduction: false
    deployCluster: ${{ parameters.deployCluster }}

Pipeline for Prod environment:

parameters:
- name: deployCluster
  type: boolean
  default: false

trigger:
  branches:
    include:
    - release/*

extends:
  template: base-pipeline.yaml
  parameters:
    isProduction: true
    deployCluster: ${{ parameters.deployCluster }}

Base pipeline:

# base-pipeline.yaml

parameters:
  - name: isProduction
    displayName: A boolean value to determine if the pipeline is for production
    type: boolean

  - name: deployCluster
    type: boolean
    default: false

variables:
  tag: '$(Build.BuildId)'
  workingDirectory: '$(System.DefaultWorkingDirectory)'

pool:
  name: Company Analytics SelfHosted
  demands:
    - agent.name -equals px-dataagent-de  

stages:
  - stage: Build
    displayName: "Build"
    # ...

  - ${{ if eq(parameters.isProduction, false) }}:
    - stage: Dev
      dependsOn: ["Build"]
      displayName: "Deploy to Dev"
      # ...

    - stage: QA
      dependsOn: ["Dev"]
      displayName: "Deploy to QA"
      # ....

  - ${{ if eq(parameters.isProduction, true) }}:
    - stage: Prod
      dependsOn: ["Build"]
      displayName: "Deploy to Prod"
      # ...

Note:

Conditions to check which stage(s) to run based on Build.SourceBranch are not needed anymore.A template expression based on parameter isProduction is used instead.