azure-devopsazure-pipelinesazure-pipelines-yaml

In Azure Devops Yaml pipeline, how to programatically add a step to a deployment


The company I'm at, has many Azure Devops YAML pipelines, spread across multiple Git repositories. These multi-stage pipelines are used to build .NET applications and to deploy them to four different stages (dev/test/accp/prod).

To reduce the amount of files being copied from one pipeline to the next, I've created a separate Git repo that contains reusable templates that the various pipelines can reference.

One of the templates is performing some cross-cutting concerns when deploying applications, and I want to extend this template to inject a preDeploy step for each jobs.deployment. To illustrate the problem, I've constructed a very contrived example consisting of 3 files. The first is a template that contains a deployment job; based on the parameter includePredeployStep a hard-coded preDeploy step will be included:

# File: deployJob.yaml
parameters:
- name: deploymentName
  type: string
- name: environment
  type: string
- name: includePredeployStep
  type: boolean

jobs:  
  - deployment: ${{parameters.deploymentName}}
    environment:
        name: ${{parameters.environment}}
        resourceType: virtualMachine
    strategy:      
      runOnce:
        deploy:
          steps:
            - powershell: | 
                Write-Host "Inside job deploy step"
        ${{if eq(parameters.includePredeployStep, 'true')}}:
          preDeploy:
            steps:
              - powershell: | 
                  Write-Host "Inside job preDeploy step"

To inject a preDeploy step, I've created another template that is heavily inspired on Microsoft's example of how to use each, this is the file that I cannot get to work:

# File: execute-deployment-jobs.yaml
parameters:
- name: deployJobs  
  type: jobList

jobs: 
- ${{ each job in parameters.deployJobs }}: 
  - ${{ each deployJobProperty in job }}:
      ${{ if and(ne(deployJobProperty.key, 'dependsOn'), ne(deployJobProperty.key,'strategy')) }}:
        ${{ deployJobProperty.key }}: ${{ deployJobProperty.value }}
      ${{ if eq(deployJobProperty.key, 'strategy')}}:        
        strategy:
          ${{if deployJobProperty.value.runOnce}}:
            runOnce: 
              ${{each runOnceProperty in deployJobProperty.value.runOnce}}: 
                ${{if ne(runOnceProperty.key, 'preDeploy')}}:
                  ${{runOnceProperty.key}}: ${{runOnceProperty.value}}                 
                preDeploy:
                  steps:
                  - powershell: | 
                      Write-Host "Inside injected preDeploy step"
                    displayName: Injected preDeploy step
                  - ${{each preDeployStep in runOnceProperty.predeploy.steps}}:
                    - ${{preDeployStep}}  

Finally, I use file 'execute-deployment-jobs' in my pipeline:

# File: pipeline.yaml
pool:
  name: Development  

stages:
- stage: Test
  jobs:
  - template: execute-deployment-jobs.yaml
    parameters:
      deployJobs:        
        - template: deployJob.yaml 
          parameters:
            deploymentName: MyApplication
            environment: Test
            includePredeployStep: false #  <--- Value 'false' does not cause a problem
        - template: deployJob.yaml 
          parameters:
            deploymentName: MyOtherApplication
            environment: Test
            includePredeployStep: true #  <--- Value 'true' causes the problem

When I validate the template, I get an error saying "'preDeploy' is already defined". The problem only arises when includePredeployStep on the very last line in the pipeline is set to true; if I change the value to false, the validation of the template succeeds and the compiled template shows the injected preDeploy step.

Any suggestions on how I can modify file 'execute-deployment-jobs.yaml' such that it will work for jobs with or without build-in preDeploy steps?

Btw, much of the code in 'execute-deployment-jobs' must be duplicated for strategy canary and rolling. Don't know if there is a quick fix to solve that.

---------- Update 1 ----------

As a clarification; what I want of 'execute-deployment-jobs' is for it to always inject a preDeploy step into any deployment jobs, regardless of whether the deployment job already has a preDeploy section. In the example pipeline, deployment of application 'MyOtherApplication' causes a compile error because it has a job-specific preDeploy section (that prints out "Inside job preDeploy step"). If I remove this job-specific preDeploy section, the template can compile and both deployment jobs have an injected preDeploy section (printing out "Inside injected preDeploy step"). What causes the compile error?


Solution

  • It seens that the attempt to nest preDeploy steps within another preDeploy section leads to conflicts.

    I tested the issue and modified the execute-deployment-jobs.yaml. It will check if preDeploy already exists and merge the steps if it does, or create a new preDeploy section if it doesn't.

    parameters:
    - name: deployJobs  
      type: jobList
    
    jobs: 
    - ${{ each job in parameters.deployJobs }}: 
      - ${{ each deployJobProperty in job }}:
          ${{ if and(ne(deployJobProperty.key, 'dependsOn'), ne(deployJobProperty.key, 'strategy')) }}:
            ${{ deployJobProperty.key }}: ${{ deployJobProperty.value }}
          ${{ if eq(deployJobProperty.key, 'strategy') }}:        
            strategy:
              runOnce:
                ${{ each runOnceProperty in deployJobProperty.value.runOnce }}:
                  ${{ if eq(runOnceProperty.key, 'preDeploy') }}:
                    preDeploy:
                      steps:
                      - powershell: |
                          Write-Host "Inside injected preDeploy step"
                        displayName: Injected preDeploy step
                      - ${{ each preDeployStep in runOnceProperty.value.steps }}:
                          ${{ preDeployStep }}
                  ${{ if ne(runOnceProperty.key, 'preDeploy') }}:
                    ${{ runOnceProperty.key }}: ${{ runOnceProperty.value }}
                ${{ if not(deployJobProperty.value.runOnce.preDeploy) }}:
                  preDeploy:
                    steps:
                    - powershell: |
                        Write-Host "Inside injected preDeploy step"
                      displayName: Injected preDeploy step
    
    

    Test result:

    test