[SOLVED] Gitlab Kubernetes Agent behind Nginx Reverse Proxy

Dear friendly developer,

I am trying to register a Gitlab Kubernetes Agent inside a Minikube with a self hosted Gitlab instance. The Gitlab instance is a dockerized Omnibus installation. It does not have any exposed ports. Instead I chose to use a nginx within the same docker network to proxy_pass requests to Gitlab.

When I deploy the agent to the cluster and the container is running, it logs theses errors:

{"level":"warn","time":"2022-02-26T00:12:59.647Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSNZ31HRVTAAD5J5700BBXH"}
{"level":"error","time":"2022-02-26T00:13:28.271Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSP040J2CRGF5WFHMEX1ACC"}

Visiting http://gitlab.local/api/v4/internal/kubernetes/agent_info results in

{
  "message": "KAS JWT authentication invalid"
}

The agent successfully connects to Gitlab when I expose the gitlab ports directly to localhost (and change the agent's kubernetes config accordingly). That is why I am quite sure that it has to be a problem with my nginx websocket configuration.

I have triple checked that the token inside the kubernetes secret for the agent matches the base64 registration token generated by Gitlab.

This is an excerpt of my docker-compose file for gitlab:

services:
  gitlab:
    image: gitlab/gitlab-ee:latest
    container_name: gitlab
    restart: always
    hostname: gitlab.local
    networks:
      - ci-cd
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.local'
        registry_external_url 'http://gitlab.local:5050'
        registry['enable'] = true
        registry['env'] = {
          "REGISTRY_HTTP_RELATIVEURLS" => true
        }
        gitlab_kas['enable'] = true
        gitlab_kas['gitlab_address'] = 'http://gitlab.local'
    volumes:
      - $GITLAB_HOME/etc:/etc/gitlab:rw
      - $GITLAB_HOME/opt:/var/opt/gitlab:rw
      - $GITLAB_HOME/log:/var/log/gitlab:rw
    shm_size: "512m"
    ulimits:
      sigpending: 62793
      nproc: 131072
      nofile: 60000
      core: 0
    sysctls:
      net.core.somaxconn: 1024

The default API path that gitlab uses for the agent websocket connection is: /-/kubernetes-agent/

This is my nginx configuration:

upstream gitlab_container {
 server gitlab;
}

upstream gitlab_registry_container {
 server gitlab:5050;
}

map $http_upgrade $connection_upgrade {
 default upgrade;
 `` close;
}

server {
 listen 80;
 listen [::]:80;

 server_name gitlab.local;

 location / {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-NginX-Proxy true;
  proxy_set_header Host $host;
  proxy_pass http://gitlab_container;
  proxy_ssl_session_reuse off;
  proxy_redirect off;
  proxy_cache_bypass $http_upgrade;
 }

 location /-/kubernetes-agent/ {
  proxy_pass http://gitlab;
  proxy_http_version 1.1;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-NginX-Proxy true;
  proxy_set_header Host $host;
  proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
  proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
  proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
  proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_cache_bypass $http_upgrade;
 }
}

server {
 listen 5050;
 listen [::]:5050;

 server_name gitlab.local;

 location / {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-NginX-Proxy true;
  proxy_set_header Host $host;
  proxy_pass http://gitlab_registry_container;
  proxy_redirect off;
  proxy_ssl_session_reuse off;
  proxy_cache_bypass $http_upgrade;
 }
}

This is the kubernetes configuration for my agent:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab-agent
  namespace: gitlab-kubernetes-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitlab-agent
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app: gitlab-agent
    spec:
      hostAliases:
        - ip: ${INTERNAL_HOST_IP}
          hostnames:
            - "gitlab.local"
      containers:
        - args:
            - --token-file=/config/token
            - --kas-address
            - ws://gitlab.local/-/kubernetes-agent/
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          image: registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable
          livenessProbe:
            httpGet:
              path: /liveness
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20
          name: agent
          readinessProbe:
            httpGet:
              path: /readiness
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          volumeMounts:
            - mountPath: /config
              name: token-volume
      serviceAccountName: gitlab-agent
      volumes:
        - name: token-volume
          secret:
            secretName: ${GITLAB_AGENT_TOKEN_NAME}

The handshake and the protocol upgrade seems to be working fine, as my nginx log shows

172.19.0.1 - - [26/Feb/2022:00:29:32 +0000] "GET /-/kubernetes-agent/ HTTP/1.1" 101 3450 "-" "gitlab-agent/v14.8.1/86d5bf7" "-"

I guess that somehow the registration token gets lost when passing through the reverse proxy. Sadly, I cannot find any technical documentation on how the authentication works in detail.

Any clue as to what I am missing is highly appreciated!