dockerubuntukubernetesjenkinstestcontainers

Docker Desktop 1node Kubernetes Jenkins share docker.sock from host into agent docker container


I have been trying to fix this few days, but I have no idea how to make it work. I have Ubuntu 22.04.5 as a host machine, there is an installed docker desktop running Kubernetes. I created Dockerfile extending official jenkins image as follows:

FROM jenkins/jenkins:2.491-jdk21

USER 0
RUN apt-get update
RUN apt-get -y install ca-certificates curl sudo
RUN install -m 0755 -d /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
RUN chmod a+r /etc/apt/keyrings/docker.asc
RUN echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
      bookworm stable" | \
      tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get -y install docker-ce-cli docker-compose-plugin

ARG DOCKER_GID=999
RUN groupadd -g ${DOCKER_GID} docker
RUN usermod -aG docker jenkins

RUN cd /var/jenkins_home
RUN git config --global --add safe.directory '*'
RUN cd /

COPY daemon.json /etc/docker/daemon.json

In a nutshell, this Dockerfile aims to add a docker client to the official Jenkins image. It works like a charm, but the crucial part is to connect this docker client with docker daemon running on host machine. The docker group from host machine is mirrored with the GID = 999 and jenkins user is added to the group.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-deployment
  namespace: devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
      serviceAccountName: jenkins-admin
      initContainers:
        - name: fix-permissions
          image: busybox
          command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home" ]
          securityContext:
            privileged: true
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
      containers:
        - name: jenkins-container
          image: myregistry/jenkins-jdk21
          imagePullPolicy: Always
          securityContext:
            privileged: true
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
            - name: docker-sock
              mountPath: /var/run/docker.sock
      imagePullSecrets:
        - name: nexus-docker-registry
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
            claimName: jenkins-pvc
        - name: docker-sock
          hostPath:
            path: "/host_mnt/var/run/docker.sock"
            type: Socket

Afterall, I am able to make it run, I see the file at /var/run/docker.sock, however when I type command docker info, then I get the following output:

Client: Docker Engine - Community
 Version:    27.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

I went through all of these forums and everything refers to problem with permissions, however, I am running it with the user (with root privileges and member of docker group) on the host machine and it is linked with Jenkins user UID = 1000 which is member of docker group. Can anyone help me?


Solution

  • I was thinking about it, and subsequently, I decided to remove Jenkins from Kubernetes and then deploy it as an individual docker container. It worked like a charm, I used the following command docker run:

    docker run -u 0 --privileged -d -v /var/run/docker.sock:/var/run/docker.sock -v /my-storage/for-jenkins-home:/var/jenkins_home -p 30180:8080 company/my-jenkins-image
    

    With this setup I can share docker daemon from the host and use it also for Jenkins agents and Testcontainers. It means 2 nested docker containers:

    jenkins-container -> jenkins agent container -> testcontainer
    

    When you want to call Ryuk from the inside of the Jenkins agent container you have to use a well-known env variable:

    TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal