spinnaker

Can you stop one Spinnaker pipeline based on a different pipeline running?

I have two pipelines as part of my deployment.

I have both pipelines set to not allow concurrent runs - you can't start a cleanup while another cleanup is going on, you can't start a deploy with another deploy already running.

However, I don't want a Deploy to start if Clean Up is running. Clean Up may incorrectly reset things out from under the Deploy and cause it to fail or, worse, incorrectly succeed.

How do I ensure Deploy only runs if Clean Up is not running and if no other Deploy is running?

UPDATE: I recognize this could all be part of one pipeline, however, in order to clarify why these are separate pipelines let me add context.

The Deploy pipeline has about 20 steps, some of which run in parallel. A sort of simplified look at it would be:

Bake and Deploy
Manifest as Canary
         |
  +------+------+
  |      |      |
Tests  Tests  Tests
  |      |      |
  +------+------+
         |
Enable Some Prod Traffic
         |
  +------+------+
  |      |      |
Tests  Tests  Tests
  |      |      |
  +------+------+
         |
Enable More Prod Traffic
         |
Automated Canary Analysis
         |
Enable More Prod Traffic
         |
Automated Canary Analysis
         |
Finish Rolling New Version
Into Production

Clean Up has about five steps, but it boils down to:

Ensure No Network Traffic Remains
Pointed at Canary Instances (Reset
Networking)
    |
Delete Any Remaining Canary Instances

In the Deploy pipeline, if any of the initial sets of parallel tests fail, I need to run that cleanup - reset the networking and kill the canary. If either of the Automated Canary Analysis runs fail, I need to run the cleanup. At the end of a successful deploy... I need to run the cleanup to make sure there's no remaining canary stuff out there.

The workflow, from what I can tell, isn't really procedural. I can't "GOTO" the cleanup. I also really, really don't want to copy/paste the set of five-ish steps into every place it could succeed or fail. I also want to make sure that if something else fails - maybe the step that enables prod traffic or something - that the cleanup runs. I don't always have a simple way to just "hook in" that I can find.

I also haven't found any sort of "on error, execute this set of steps" or "on success, execute this set of steps" other than a separate pipeline that can trigger based on those criteria.

My ideal solution would be, in pseudocode:

try {
  // do the deploy steps
}
finally {
  // whether success or fail
  // run the clean up steps
}

However, given the limitation I see with being unable to have try/finally in the pipeline, I'm triggering the Clean Up pipeline whenever Deploy completes (success or fail). However, there's nothing I can see that can "block" Deploy from starting while Clean Up is still running, so if there is a set of backed-up changes that are all waiting for deployment, the cleanup stuff may clean things out from under me.

Solution

  • I don't think Spinnaker has that specific feature, but one thing you could do is have your Deploy and Clean Up stages all part of the same pipeline, running in parallel, and do not allow concurrent runs. See example in screenshot below.

    This way if Deploy or Clean Up stages are already running, the pipeline should not start. If your canary fails, the clean up should still run. If your canary passes, the clean up will still run, and deployment will occur.

    enter image description here