Gitlab CI Runner fails to push image to container registry - unexpected HTTP status: 200 OK

Introduction to problem

Hi, i deployed Gitlab on Rocky Linux 8 VM and Gitlab-runner on RHEL 9 VM and custom self-signed CA.

Gitlab is up and running and jobs can be picked up with the runners successfully.

I want to create a custom docker image and push it to my container registry that is hosted in Gitlab (not external registry).

Dockerfile in Gitlab.

.gitlab-ci.yml

build image:
image: docker:20.10.17
cache: []
services:
    - name: docker:20.10.17-dind
script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
    - docker build -t $CI_REGISTRY_IMAGE .
    - echo $GITLAB_REGISTRY_API_URL
    - echo $CI_REGISTRY_IMAGE
    - docker push $CI_REGISTRY_IMAGE

Job fails

When the job gets triggered with the above .gitlab-ci.yml file then it fails by retrying to push the image and at the end with the error received unexpected HTTP status: 200 OK

Output of some relevant information/configuration to help debug or reproduce the problem

on Server hosting Gitlab (Rocky 8)

/etc/gitlab/gitlab.rb

    [root@magnaslab nginx]# grep -v '^#\|^$' /etc/gitlab/gitlab.rb
external_url 'https://gitlab.magnas.com'
gitlab_rails['gitlab_default_theme'] = 1
gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = false
 gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
   main: # 'main' is the GitLab 'provider ID' of this LDAP server
     label: 'magnas'
     host: 'DC01.magnas.com'
     port: 389
     uid: 'sAMAccountName'
     encryption: 'plain' # "start_tls" or "simple_tls" or "plain"
     active_directory: true
     allow_username_or_email_login: true
     lowercase_usernames: true
     base: ''
     group_base: ''
     admin_group: ''
     sync_ssh_keys: false
EOS
registry_external_url 'https://gitlab.magnas.com'
registry['env'] = {
 "REGISTRY_HTTP_RELATIVEURLS" => true
}
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.magnas.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.magnas.com.key"
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.magnas.com"
gitlab_rails['registry_port'] = "80"
registry['enable'] = true
registry['username'] = "registry"
registry['env'] = {
   'SSL_CERT_DIR' => "/opt/gitlab/embedded/ssl/certs/"
}
registry['autoredirect'] = true
nginx['enable'] = true
nginx['client_max_body_size'] = '250m'
nginx['redirect_http_to_https'] = true
nginx['redirect_http_to_https_port'] = 80
nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/gitlab.magnas.com.crt"
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.magnas.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.magnas.com.key"
nginx['listen_addresses'] = ["0.0.0.0", "[::]"]
nginx['listen_port'] = 443
registry_nginx['enable'] = true
registry_nginx['listen_https'] = true
registry_nginx['redirect_http_to_https'] = true
letsencrypt['enable'] = false # GitLab 10.5 and 10.6 require this option
external_url "https://gitlab.magnas.com"   # Must use https protocol
letsencrypt['contact_emails'] = ['xxxx@magnas.com' 'xxx@magnas.com'] # This should be an array of email addresses to add as contacts

/var/opt/gitlab/nginx/conf/gitlab-http.conf

        ## Redirects all HTTP traffic to the HTTPS host
    server {
      listen 0.0.0.0:80;
      listen [::]:80;
    
      server_name gitlab.magnas.com;
      server_tokens off; ## Don't show the nginx version number, a security best practice
    
    
      location / {
        return 301 https://gitlab.magnas.com:443$request_uri;
      }
    
      # health checks configuration
      include /var/opt/gitlab/nginx/conf/gitlab-health.conf;
    
      access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
      error_log   /var/log/gitlab/nginx/gitlab_error.log error;
    }

server {
  listen 0.0.0.0:443 ssl http2;


  listen [::]:443 ssl http2;


  server_name gitlab.magnas.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  ## Increase this if you want to upload large attachments
  ## Or if you want to accept large git objects over http
  client_max_body_size 250m;

  ## Strong SSL Security
  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
  ssl_certificate /etc/gitlab/ssl/gitlab.magnas.com.crt;
  ssl_certificate_key /etc/gitlab/ssl/gitlab.magnas.com.key;
  ssl_client_certificate /etc/gitlab/ssl/gitlab.magnas.com.crt;

  # GitLab needs backwards compatible ciphers to retain compatibility with Java IDEs
  ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AES256-GCM-SHA384';
  ssl_protocols  TLSv1.2 TLSv1.3;
  ssl_prefer_server_ciphers off;
  ssl_session_cache  shared:SSL:10m;
  ssl_session_tickets off;
  ssl_session_timeout  1d;


  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  ## HSTS Config
  ## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
  add_header Strict-Transport-Security "max-age=63072000";

  # Rails sets a default policy of strict-origin-when-cross-origin, so
  # hide that and just send the one we've configured for nginx
  proxy_hide_header Referrer-Policy;
  add_header Referrer-Policy strict-origin-when-cross-origin;

## Individual nginx logs for this GitLab vhost
  access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_error.log error;

  if ($http_host = "") {
    set $http_host_with_default "gitlab.magnas.com";
  }

  if ($http_host != "") {
    set $http_host_with_default $http_host;
  }

  gzip on;
  gzip_static on;
  gzip_comp_level 2;
  gzip_http_version 1.1;
  gzip_vary on;
  gzip_disable "msie6";
  gzip_min_length 250;
  gzip_proxied no-cache no-store private expired auth;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;

  ## https://github.com/gitlabhq/gitlabhq/issues/694
  ## Some requests take more than 30 seconds.
  proxy_read_timeout      3600;
  proxy_connect_timeout   300;
  proxy_redirect          off;
  proxy_http_version 1.1;

  proxy_set_header Host $http_host_with_default;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_set_header X-Forwarded-Proto https;
  proxy_set_header X-Forwarded-Ssl on;

  location ~ (/api/v\d/jobs/\d+/artifacts$|\.git/git-receive-pack$|\.git/gitlab-lfs/objects|\.git/info/lfs/objects/batch$) {
    proxy_cache off;
    proxy_pass http://gitlab-workhorse;
    proxy_request_buffering off;
  }

  location /-/grafana/ {
    proxy_pass http://localhost:3000/;
  }

  location = /-/kubernetes-agent/ {
    proxy_pass http://localhost:8150/;
  }

  location /-/kubernetes-agent/k8s-proxy/ {
    proxy_pass http://localhost:8154/;
  }

  # health checks configuration
  include /var/opt/gitlab/nginx/conf/gitlab-health.conf;

  location / {
    proxy_cache off;
    proxy_pass  http://gitlab-workhorse;
  }

  location /assets {
    add_header X-Content-Type-Options nosniff;
    proxy_cache gitlab;
    proxy_pass  http://gitlab-workhorse;
  }

  error_page 404 /404.html;
  error_page 500 /500.html;
  error_page 502 /502.html;
  location ~ ^/(404|500|502)(-custom)?\.html$ {
    root /opt/gitlab/embedded/service/gitlab-rails/public;
    internal;
  }

/var/log/gitlab/nginx/error.log

/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:41 [emerg] 3484915#0: no "ssl_certificate" is defined for the "listen ... ssl" directive in /var/opt/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:42 [emerg] 3484932#0: no "ssl_certificate" is defined for the "listen ... ssl" directive in /var/opt/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:43 [emerg] 3485005#0: no "ssl_certificate" is defined for the "listen ... ssl" directive in /var/opt/gitlab/nginx/conf/gitlab-http.conf:34
2022/08/02 08:58:44 [warn] 3485019#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 08:58:44 [warn] 3485019#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 10:16:56 [warn] 3494210#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 10:24:13 [warn] 3495595#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 10:24:13 [warn] 3495595#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 11:31:36 [warn] 3504813#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 11:31:36 [warn] 3504813#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 11:36:57 [warn] 3506143#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 11:36:57 [warn] 3506143#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored
2022/08/02 11:37:02 [warn] 3506221#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:80, ignored
2022/08/02 11:37:02 [warn] 3506221#0: conflicting server name "gitlab.magnas.com" on 0.0.0.0:443, ignored

on Server hosting Gitlab-runner (RHEL 9)

/etc/gitlab-runner/conf.toml

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "rhel-runner-2"
  url = "https://gitlab.magnas.com"
  token = "rxF8jJtMTJdBgayB-Vbe"
  tls-cert-file = "/etc/gitlab-runner/certs/gitlab.magnas.com.crt"
  tls-key-file = "/etc/gitlab-runner/certs/gitlab.magnas.com.key"
  executor = "docker"
  clone_url = "http://10.30.36.240"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    insecure = true
    image = "ruby:2.7"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0

/etc/docker/daemon.json

I was getting HTTPS error "Unknown Certificate" so i found a workaround and added this file.

{"insecure-registries" : ["gitlab.magnas.com:80"]}

/etc/default/docker

DOCKER_OPTS="--config-file=/etc/docker/daemon.json"

What troubleshooting steps are yet to be taken? Any help is well much appreciated.

Thanks!

From logs that you've provided, it looks like that SSL certificate is not properly configured on your Gitlab instance.

First thing I would try is to setup let's encrypt SSL certificate instead of your gitlab.magnas.com certificate just to get your CI/CD pipeline to work properly without any workaround. See the documentation here. Gitlab has a nice out-of-the-box support for let's encrypt certificates.

From my opinion, let's encrypt SSL cert is good solution for self-hosted Gitlab instance behind VPN.

Just to mention, I had a similar problem with custom SSL certificate on Gitlab instance. The problem was that my /etc/gitlab-runner/certs/custom.crt was missing root CA certificates. This file needs to contain your certificate + root CA certificates.

Hope this would help.