dockergitlabgitlab-cigitlab-ci-runnerroot-certificate

How to make GitLab Runner in Docker see a custom CA Root certificate


I have installed and configured:

  1. an on-premises GitLab Omnibus on ServerA running on HTTPS
  2. an on-premises GitLab-Runner installed as Docker Service in ServerB

ServerA certificate is generated by a custom CA Root

The Configuration

I've have put the CA Root Certificate on ServerB:

/srv/gitlab-runner/config/certs/ca.crt

Installed the Runner on ServerB as described in Run GitLab Runner in a container - Docker image installation and configuration:

docker run -d --name gitlab-runner --restart always \
           -v /srv/gitlab-runner/config:/etc/gitlab-runner \
           -v /var/run/docker.sock:/var/run/docker.sock \
           gitlab/gitlab-runner:latest

Registered the Runner as described in Registering Runners - One-line registration command:

docker run --rm -t -i 
            -v /srv/gitlab-runner/config:/etc/gitlab-runner 
           --name gitlab-docker-runner gitlab/gitlab-runner register \
           --non-interactive \
           --executor "docker" \
           --docker-image alpine:latest \
           --url "https://MY_PRIVATE_REPO_URL_HERE/" \
           --registration-token "MY_PRIVATE_TOKEN_HERE" \
           --description "MyDockerServer-Runner" \
           --tag-list "TAG_1,TAG_2,TAG_3" \
           --run-untagged \
           --locked="false"

This command gave the following output:

Updating CA certificates...
Runtime platform arch=amd64 os=linux pid=5 revision=cf91d5e1 version=11.4.2
Running in system-mode.

Registering runner... succeeded runner=8UtcUXCY
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

I checked with

$ docker exec -it gitlab-runner bash 

and once in the container with

$ awk -v cmd='openssl x509 -noout -subject' '
/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

and the custom CA root is correctly there.

The Problem

When running Gitlab-Runner from GitLab-CI, the pipeline fails miserably telling me that:

$ git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git


Cloning into 'My-Project.wiki'...


fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@ServerA/foo/bar/My-Project.wiki.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none


ERROR: Job failed: exit code 1

It does not recognize the Issuer (my custom CA Root), but according to The self-signed certificates or custom Certification Authorities, point n.1, it should out-of-the-box:

Default: GitLab Runner reads system certificate store and verifies the GitLab server against the CA’s stored in system.

I've then tried the solution from point n.3, editing

/srv/gitlab-runner/config/config.toml:

and adding:

[[runners]]
tls-ca-file = "/srv/gitlab-runner/config/certs/ca.crt"

But it still doesn't work.

How can I make Gitlab Runner read the CA Root certificate?


Solution

  • While I've still not got why it doesn't work out-of-the-box, I've found the Egg of Columbus:

    Gitlab-Runner configuration:

    [[runners]]
      name = "MyDockerServer-Runner"
      url = "https://MY_PRIVATE_REPO_URL_HERE/"
      token = "MY_TOKEN_HERE"
      executor = "docker"
      ...
      [runners.docker]
        image = "ubuntu:latest"
    
      # The trick is the following:
        volumes = ["/cache","/srv/gitlab-runner/config:/etc/gitlab-runner"]
        ...
    

    Gitlab-ci.yml pipeline:

    MyJob:
        image: ubuntu:latest
    
        script:
          - awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
          - git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git
          - wget -O foo.png https://ServerA/foo/bar/foo.png 
    
        before_script:
          - apt-get update -y >/dev/null
          - apt-get install -y apt-utils dialog >/dev/null
          - apt-get install -y git >/dev/null
          - apt-get install -y wget >/dev/null
    
        # The trick is the following:
          - cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
          - update-ca-certificates
    

    That's it:

    And everything will work as expected: git clone, wget https, etc...

    A great workaround, until someone at GitLab will fix it or explain me where I'm wrong (be my guest!)