jenkinsjenkins-pipelinejenkins-groovy

How can I async Jenkins parallel stages?


Jenkins can run parallel stages and they can execute simultaneously and independently of each other. How can I start a stage after another stage has been completed in another parallel pipeline?

For example, there are two parallel pipelines. Stage B be started after stage A. And Stage C must be run after stage B.

-------stage A--------stage C---------------------
--------------stage B-----------------------------

Please show me a Declarative Pipeline example. Thanks.


Solution

  • For example, there are two parallel pipelines. Stage B be started after stage A. And Stage C must be run after stage B.

    What you describe is the sequential execution, not parallel. I assume you meant that the stage B could be triggered while the stage A is already running.

    How can I start a stage after another stage has been completed in another parallel pipeline?

    In short - you can't. All the stages of a particular pipeline are processed within that pipeline only.

    You can simulate that with the whole pipelines (not stages) using waitForBuild step in your stage C. But it will result in wasting the executor polling the build status:

    // Pipeline1 located in your/project/folder
    pipeline {
      agent any
      stages {
        stage('A') {
          steps {
            println 'I am stage $STAGE_NAME'
          }
        }
        stage('C') {
          steps {
            waitForBuild: 'your/project/folder/Pipeline1'
            println 'I am stage $STAGE_NAME'
          }
        }
      }
    }
    
    // Pipeline2 located in your/project/folder
    pipeline {
      agent any
      stages {
        stage('B') {
          steps {
            println 'I am stage $STAGE_NAME'
          }
        }
      }
    }
    

    A better option would probably be to split the Pipeline1 and move the stage C to, say, Pipeline3. That way you can make use of the build step:

    // Pipeline1 located in your/project/folder
    pipeline {
      agent any
      stages {
        stage('A') {
          steps {
            println 'I am stage $STAGE_NAME'
          }
        }
      }
    }
    
    // Pipeline2 located in your/project/folder
    pipeline {
      agent any
      stages {
        stage('B') {
          steps {
            println 'I am stage $STAGE_NAME'
            build job: 'your/project/folder/Pipeline3',
              wait: false,
              propagate: false
          }
        }
      }
    }
    
    // Pipeline3 located in your/project/folder
    pipeline {
      agent any
      stages {
        stage('C') {
          steps {
            println 'I am stage $STAGE_NAME'
          }
        }
      }
    }
    

    You can also try to simulate that using the Lock Resource plugin like described in https://stackoverflow.com/a/73059059/16417367 but in a slightly different way so that the executor doesn't get wasted by the lock step. But the problem here will be that the stages B and C will mutually lock each other so you will not be able to run another instance (like branch) of the pipeline containing the stage B until the already running stage C is finished:

    // Pipeline1 located in your/project/folder
    pipeline {
      agent any
      stages {
        stage('A') {
          steps {
            println 'I am stage $STAGE_NAME'
          }
        }
        stage('C') {
          options {
            lock('my-lock')
          }
          steps {
            println 'I am stage $STAGE_NAME'
          }
        }
      }
    }
    
    // Pipeline2 located in your/project/folder
    pipeline {
      agent any
      stages {
        stage('B') {
          options {
            lock('my-lock')
          }
          steps {
            println 'I am stage $STAGE_NAME'
          }
        }
      }
    }
    

    So my recommendation would be to redesign your pipelines to avoid the dependencies between them. Running some stages in parallel and some in sequence within the same pipeline might help.