dockergitlabgitlab-cigitlab-ci-runnerminio

GitLab Runner Docker Executor and S3 cache


even with the multiple posts over the internet, I can't figure out how to make my GitLab-Runner working...

I'm using GitLab CE 17.0.0 + 2 GitLab Runners 17.0.0, one hosted on an AlmaLinux 8 server and one hosted on a Windows 11 computer.

Everything worked fine but I would like to set up the Shared Cache.

I've set up a MinIO server, hosted on an AlmaLinux 8. GitLab Container Registry is working well with my MinIO server.

Now I would like to set up my GitLab Runners. Both are using Docker executor. Config files are very similar:

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner-windows"
  url = "https://gitlab-url"
  id = 19
  token = "..."
  executor = "docker"

  [runners.cache]
    Type = "s3"
    Shared = true

    [runners.cache.s3]
      ServerAddress = "minio-url:9000"
      AccessKey = "..."
      SecretKey = "..."
      BucketName = "gitlab-ci-cache"
      BucketLocation = "eu-east-1"

  [runners.docker]
    tls_verify = false
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["gitlab-pipeline-cache:/cache"]
    shm_size = 0

This configuration is not using my MinIO server as cache server. A Docker volume "gitlab-pipeline-cache" is created and used.

From the runner hosts, if I use the MinIO client, I success to connect to my MinIO server, to upload files, etc. It's not a network issue.

Thank you!


EDIT 1: Add my .gitlab-ci.yml content + GitLab CI job output

.gitlab-ci.yml content

workflow:
  rules:
    - if: $CI_COMMIT_TAG != null
    - if: $CI_PIPELINE_SOURCE == "web"

variables:
  CACHE_DIR: /cache/$CI_PROJECT_ROOT_NAMESPACE/$CI_PROJECT_NAME/$CI_PIPELINE_ID

stages:
  - .pre
  - touch

create_cache_dir:
  stage: .pre
  tags:
    - runner-almalinux8
  image: alpine:latest
  script:
    - mkdir --parents $CACHE_DIR/

create_file:
  stage: touch
  tags:
    - runner-windows
  image: alpine:latest
  script:
    - touch $CACHE_DIR/test_create_file.txt

Job output (.pre)

Running with gitlab-runner 17.0.0 (44feccdf)
  on runner-almalinux8 -_vLbzjNv, system ID: s_4f5c9ad29d6f
Preparing the "docker" executor
00:02
Using Docker executor with image alpine:latest ...
Pulling docker image alpine:latest ...
Using docker image sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd for alpine:latest with digest alpine@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b ...
Preparing environment
00:01
Running on runner--vlbzjnv-project-116-concurrent-0 via runner-almalinux8...
Getting source from Git repository
00:01
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/<group>/<project>/.git/
Checking out 456a298a as detached HEAD (ref is 1.0.0-rc1)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
Using docker image sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd for alpine:latest with digest alpine@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b ...
$ mkdir --parents $CACHE_DIR/
Cleaning up project directory and file based variables
00:01
Job succeeded

Job output (touch)

Running with gitlab-runner 17.0.0 (44feccdf)
  on runner-windows cc5wbtykV, system ID: s_4f5c9ad29d6f
Preparing the "docker" executor
00:08
Using Docker executor with image alpine:latest ...
Using helper image:  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.0.0
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.0.0 ...
Using docker image sha256:cb32fd9b1984b484e20e7b6806bd3a0ef5304abee2f0c64d5b38e1234c2a7bf5 for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.0.0 with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:aa094d2434e42a61215a64dfb50fb9b9dc29d81e4d708c1c896d0818a5d6f873 ...
Pulling docker image alpine:latest ...
Using docker image sha256:1d34ffeaf190be23d3de5a8de0a436676b758f48f835c3a2d4768b798c15a7f1 for alpine:latest with digest alpine@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd ...
Preparing environment
00:01
Running on runner-cc5wbtykv-project-326-concurrent-0 via runner-windows...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/<group>/<project>/.git/
Created fresh repository.
Checking out 456a298a as detached HEAD (ref is 1.0.0-rc1)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
Using docker image sha256:1d34ffeaf190be23d3de5a8de0a436676b758f48f835c3a2d4768b798c15a7f1 for alpine:latest with digest alpine@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd ...
$ touch $CACHE_DIR/test_create_file.txt
touch: /cache/<group>/<project>/3089/test_create_file.txt: No such file or directory
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Solution

  • The cache location is where GitLab will store cache bundles. To actually cache things you need to declare cacheable items:

    default:
      cache:
        paths:
        - test_create_file.txt
    
    job1:
      stage: build
      script:
      - echo "hello world" > test_create_file.txt
    
    job2:
      stage: test
      script:
      - cat test_crete_file.txt