I am looking for a way to clean up the runner after a job has been cancelled in GitLab. The reason is we often have to cancel running jobs because the runner is sometimes stuck in the test pipeline and I can imagine a couple of other scenarios where you would want to cancel a job and have a clean up script run after. I am looking for something like after_script
but just in the case when a job was cancelled.
I checked the GitLab keyword reference but could not find what I need.
The following part of my gitlab-ci.yaml
shows the test stage which I would like to gracefully cancel by calling docker-compose down
when the job was cancelled.
I am using a single gitlab-runner. Also, I don't use dind.
test_stage:
stage: test
only:
- master
- tags
- merge_requests
image: registry.gitlab.com/xxxxxxxxxxxxxxxxxxx
variables:
HEADLESS: "true"
script:
- docker login -u="xxxx" -p="${QUAY_IO_PASSWORD}" quay.io
- npm install
- npm run test
- npm install wait-on
- cd example
- docker-compose up --force-recreate --abort-on-container-exit --build traefik frontend &
- cd ..
- apt install -y iproute2
- export DOCKER_HOST_IP=$( /sbin/ip route|awk '/default/ { print $3 }' )
- echo "Waiting for ${DOCKER_HOST_IP}/xxxt"
- ./node_modules/.bin/wait-on "http://${DOCKER_HOST_IP}/xxx" && export BASE_URL=http://${DOCKER_HOST_IP} && npx codeceptjs run-workers 2
- cd example
- docker-compose down
after_script:
- cd example && docker-compose down
artifacts:
when: always
paths:
- /builds/project/tests/output/
retry:
max: 2
when: always
tags: [custom-runner]
Unfortunately this is not currently possible in GitLab. There have been several tickets opened in their repos, with this one being the most up-to-date.
As of the day that I'm posting this (September 27, 2022), there have been at least 14 missed deliverables for this. GitLab continues to say it's coming, but has never delivered it in the six years that this ticket has been open.
There are mitigations as far as automatic job cancelling, but unfortunately that will not help in your case.
Based on your use case, I can think of two different solutions:
Settings -> CI/CD -> General Pipelines -> Timeout
)Neither of these solutions are as robust as if GitLab themselves implemented a solution, but they can at least prevent you from having a job hang for an eternity and clogging up everything else in the pipeline.