I have a GitLab instance and a small kubernetes cluster. There is a certificate wildcard we are using both for our GitLab instance and for internal docker repository: *.ourdomain.corp. I decided to deploy some runners in our k8s cluster and installed them via Helm. Initially runner rejected our wildcard due to x509 issue and was unable to register, but I added a secret with our CA bundle and it worked like a charm (certsSecretName in chart values). But when I tried to use our custom image for build from private repository (via pointing it in config.toml), runner reported that it can't pull an image:
WARNING: Failed to pull image with policy "": image pull failed: rpc error: code = Unknown desc = pinging container registry nexus.ourdomain.corp:5556: Get "https://nexus.ourdomain.corp:5556/v2/": x509: certificate signed by unknown authority.
This cause CI job to fail. Here is the very begin of values.yaml:
image:
registry: registry.gitlab.com
image: gitlab-org/gitlab-runner
I think I need somehow pass our CA certificate bundle there in trusted certs location, but the user available in the shell don't have permissions. How I can edit values.yaml to make my runner trust to this certificate? Or any other options are there?
P.S.1 curl to our gitlab from runner shell returns x509 however runner can get tasks for images from public registries. P.S.2 Gitlab version 15.1.2, Runner manifest for 15.1.0 (helm chart gitlab/gitlab-runner version 0.42.0).
I tried to pass some options in config.toml template file from chart (ca_file, tls_file). For second option my runner lost connection with Gitlab instance. My certificate actually in the container in /home/gitlab-runner/.gitlab-runner/certs/gitlab.ourdomain.corp.crt. I tried to copy it in usr/local/share/ca-certificates/ but got permission denied error. As well as for update-ca-certificates command. I tried to map a volume formed from a secret file to /etc/gitlab-runner/certs/ca.crt and got permission denied error. Tried to break down the documentation and found that I need to set before_step with copying this crt and updating CA certificates. I tried it, but in the very beginning the Runner tries to pull the image first and fails bue to x509 error. I have no Idea what I can try more. I believe this is a prctical case to have a gitlab runner in kubernetes with pulling images from privat registries with certificates which require CA bundle.
EDIT: Found out that I had to restart k3s so that the kubelet then had a refreshed cache of certificates. Basically I ran an update-certificates command in my linux distro and then neglected to restart k3s. Once I did that I stopped getting x509 errors from the gitlab-runner executor.
This is not intended as an answer per se... I think what is happening is the ca.crt that you mount through the config.toml is mounted with root:root ownership and dwrx--------- permissions. When the entrypoint script that runs update-ca in the container kicks off it runs as the non-root user. This is by design from what I can tell and the user that runs the script (gitlab-runner uid 100 in my case) cannot access the ca.crt mounted at /etc/gitlab-runner/certs. I have adjusted parts of this values.yaml for the Helm chart here
## Configure securitycontext for the main container
## ref: http://kubernetes.io/docs/user-guide/security-context/
##
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false
runAsNonRoot: true
privileged: false
capabilities:
drop: ["ALL"]
## Configure securitycontext valid for the whole pod
## ref: http://kubernetes.io/docs/user-guide/security-context/
##
podSecurityContext:
runAsUser: 100
# runAsGroup: 65533
fsGroup: 65533
# supplementalGroups: [65533]
fsGroupChangePolicy: "OnRootMismatch" <--- added this based on the security context instructions at kubernetes.io documentation but didnt work.
Having the same problem myself so I will probably see if Gitlab ppl can help