I had a version of the following pipeline that ran as a single job, and all in this took under 6 minutes. I then updated the pipeline to be broken up into separate Jobs to make it easier to know which has failed, and now the total pipeline is 15+ minutes. I can only assume each Job is considered its own pipeline, with tear up/down process which is taking a long time across four different jobs. I'm looking for advice on how to refactor this GitLab CI to get back to my original 6 or less length of time:
image: cirrusci/flutter:stable
before_script:
- flutter pub get
- flutter clean
- flutter --version
stages:
- build-aot
- analyze
- format-check
- test-on-machine-with-coverage
build-aot:
stage: build-aot
script:
- flutter build aot
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
analyze:
stage: analyze
script:
- flutter analyze
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
format-check:
stage: format-check
script:
- flutter format --set-exit-if-changed lib test
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
test-on-machine-with-coverage:
stage: test-on-machine-with-coverage
script:
- flutter pub global activate junitreport
- export PATH="$PATH":"$HOME/.pub-cache/bin"
- flutter test --machine | tojunit -o report.xml
- flutter test --coverage ./lib
- lcov -r coverage/lcov.info '*/__test*__/*' -o coverage/lcov_cleaned.info
- genhtml coverage/lcov_cleaned.info --output=coverage
artifacts:
when: always
paths:
- rspec.xml
- coverage
reports:
junit:
- report.xml
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
A pipeline is commonly defined as a series of jobs (According to Gitlab's documentation), so in order to properly script your repository's CI/CD where splitting your jobs functions keeps a similar performance to when they were together, a common practice is to implement caching between stages.
Caching allows you to properly isolate each job runtime environment while simultaneously optimizing speed and bandwidth as it avoids downloading or building the same files over multiple times.
Now let's take a look at your gitlab-ci.yml
code, as to implement caching for optimizing your pipeline execution time. Flutter's documentation says a global cache is created once you install each package, and therefore you only have to install once per build, and it is located at .pub-cache
:
image: cirrusci/flutter:stable
variables:
PUB_CACHE: $CI_PROJECT_DIR/.pub-cache #https://docs.gitlab.com/ee/ci/yaml/README.html#cachepaths
cache:
key: ${CI_COMMIT_REF_SLUG} #Only use one cache per whole pipeline
paths:
- .pub-cache/
before_script:
- flutter pub cache --all #https://dart.dev/tools/pub/cmd/pub-cache
- flutter --version
stages:
- build-aot
- analyze
- format-check
- test-on-machine-with-coverage
build-aot:
stage: build-aot
script:
- flutter build aot
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
analyze:
stage: analyze
script:
- flutter analyze
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
format-check:
stage: format-check
script:
- flutter format --set-exit-if-changed lib test
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
test-on-machine-with-coverage:
stage: test-on-machine-with-coverage
script:
- flutter pub global activate junitreport
- export PATH="$PATH":"$HOME/.pub-cache/bin"
- flutter test --machine | tojunit -o report.xml
- flutter test --coverage ./lib
- lcov -r coverage/lcov.info '*/__test*__/*' -o coverage/lcov_cleaned.info
- genhtml coverage/lcov_cleaned.info --output=coverage
artifacts:
when: always
paths:
- rspec.xml
- coverage
reports:
junit:
- report.xml
only:
- master
- merge_requests
except:
variables:
- $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != "master"
This will still download your packages every new series of pipelines (when committing a change), removing the key:
flag should keep the cache until you manually erase it on Gitlab's interface.