I'm creating a pipeline to test helm charts by linting and checking the chart, packing the helm, installing them on a k8s cluster, run API tests on it and delete everything in the k8s cluster afterwards. I'm currently running into to sub-optimal pipeline flows with the clean-up job. My pipeline is as follows:
stages:
- helm-checks
- helm-package
- build-k8s-test-env
- tests
- clean-up-k8s-test-env
helm-lint-template:
stage: helm-checks
polaris:
stage: helm-checks
kubescape:
stage: helm-checks
helm-package:
stage: helm-package
build-k8s-test-environment:
stage: build-k8s-test-env
script:
- kubectl create ns $NAMESPACE
- helm install
hurl-tests:
stage: tests
clean-up-k8s-test-environment:
stage: clean-up-k8s-test-env
- kubectl delete all --all -n $NAMESPACE
I want my pipeline to always clean-up the created k8s environment with the clean-up-k8s-test-environment, but this should only run if the build-k8s-test-environment has run regardless if its successful or not. But the clean-up-k8s-test-environment should only run after the jobs in the tests stage have finished, in case the build was successful. Also, it should not run if the jobs in earlier stages like the helm-checks or helm-package stage have failed.
I've tried using
clean-up-k8s-test-environment:
stage: clean-up-k8s-test-env
needs:
- build-k8s-test-environment
- hurl-tests
when: always
But in this example, the clean-up-k8s-test-environment job runs when there is an failure in helm-checks or helm-package, and therefore the clean-up job will also fail as the build-k8s-test-environment will not have run. It seems that if the jobs in the needs have not run, Gitlab falls back to the when: always
Does anyone know a more optimal way I can implement this?
clean-up-k8s-test-environment: stage: clean-up-k8s-test-env needs: - build-k8s-test-environment - hurl-tests when: always
@ha36d Thank you for your suggestion. Based on your tips I created the following gitlab-ci:
stages:
- helm-checks
- helm-package
- tests
helm-lint-template:
stage: helm-checks
polaris:
stage: helm-checks
kubescape:
stage: helm-checks
helm-package:
stage: helm-package
build-k8s-test-environment:
stage: tests
script:
- kubectl create ns $NAMESPACE
- helm install
environment:
name: review/$CI_COMMIT_REF_NAME
on_stop: stop_review
auto_stop_in: 1 hour
hurl-tests:
stage: tests
needs:
- build-k8s-test-environment
clean-up-k8s-test-environment:
stage: tests
- kubectl delete all --all -n $NAMESPACE
needs:
- build-k8s-test-environment
- hurl-tests
environment:
name: review/$CI_COMMIT_REF_NAME
action: stop
What I've added is the needs to the clean-up-k8s-test-environment as this job should run after the other jobs in the same stage have finished. I removed the manual part, as I want this to be completely automatic to not hamper development time where the engineer has to manually delete the environment every time. Also adjusted the environment name purely on personal preferences.
This setup indeed causes that a failure in helm-checks does indeed not trigger the tests stage, therefore not running the clean-up-k8s-test-environment job. The downside is that a failure in the hurl-tests job will not trigger the clean-up-k8s-test-environment. I experimented with the on_failure: true setting in the hurl-tests job, but that allows merge requests to be merged with failed tests in it.
For now I've set my mind to change my setup to not delete a feature environment in every pipeline, but only delete the environment if a feature branch is merged. I will create a Kubernetes namespace one time only and check and skip the creation of it, if it already exists. Will do the same with the helm chart, if it's already in the environment, delete it before building it again. All in the build job.
This is not the ideal setup, but will do for now.
So now I have
build-k8s-test-environment:
stage: tests
script:
- kubectl delete all --all -n $KUBERNETES_NAMESPACE --ignore-not-found
- kubectl delete ns $KUBERNETES_NAMESPACE --ignore-not-found
- helm install --namespace $KUBERNETES_NAMESPACE --create-namespace