gitdockersshgitlab-ce

No connection to self-hosted Docker-Gitlab Instance possible


I have a self-hosted Docker-Gitlab (v.16.8 currently) and it worked for some years. Then, a year ago I disabled it, recently enabled it again and updated it. The Gitlab webpage, the CI jobs, the Container Registry and all that works. What does not work, though, is my ability to interact with the server via Git through SSH, e.g. git clone ssh://git@INTERNETIP:10022/repo.git

Extended error log when connecting from my machine to the Docker-Gitlab instance hosted on an Ubuntu LTS (5.15.0-91) server

OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/christian/.ssh/config line 1: Applying options for myurl.tld
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug2: resolving "myurl.tld" port 10022
debug3: resolve_host: lookup myurl.tld:10022
debug3: ssh_connect_direct: entering
debug1: Connecting to myurl.tld [externalip] port 10022.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: connect to address externalip port 10022: Connection refused
ssh: connect to host myurl.tld port 10022: Connection refused
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The setup is the following:

external_url 'https://myurl.tld'
gitlab_rails['gitlab_shell_ssh_port'] = 10022
gitlab_sshd['enable'] = false # also tried with true (which is the new Gitlab-made SSH server)
#gitlab_sshd['listen_address'] = '[::]:22' # also tried 10022 here
docker run \
--detach \
--name gitlab \
--restart always \
-p 10022:22 -p 10080:80 -p 10443:443 -p 4567:4567 \
gitlab/gitlab-ce:latest
"Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        ...
}
"Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "127.0.0.1",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
# nginx.conf
server {
       list INTERNETIP:443 ssl http2;
       server_name myurl.tld;
...
       location ~ ^/.* {
                proxy_pass http://0.0.0.0:10443;
                proxy_set_header Host              $host;
                proxy_set_header X-Real-IP         $remote_addr;
                proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
...
Public key for /etc/ssh/ssh_host_rsa_key does not match private key
debug1: private host key #0: ssh-rsa SHA256:xxx
Public key for /etc/ssh/ssh_host_ecdsa_key does not match private key
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:xxx
Public key for /etc/ssh/ssh_host_ed25519_key does not match private key
debug1: private host key #2: ssh-ed25519 SHA256:xxx

I have tracked the issue a bit:

How can I further test/debug this matter and find out what stops me from connecting to the Git server as I did a year ago?

EDIT: Here are the current iptables rules, which were auto-generated by docker

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N f2b-ssh
-A INPUT -p tcp -m tcp --dport 22 -j f2b-ssh
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 4567 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A f2b-ssh -j RETURN

Solution

  • After some more inspection and tests, especially with the firewall, it dawned on me: it is the port mapping. Somewhen back, it seems the default behavior of docker to map ports has changed?! My above script with -p 10022:22 maps the external port 10022 to the internal port 22, yet only on the interface 127.0.0.1 when I issue a docker ps. This prevents me from accessing the system from the outside via port 10022. Then the question is, why does the webinterface on port 10443 work? Well, for this I had a reverse proxy entry set up, so the HTTP/HTTPS calls are actually coming from 127.0.0.1 due to the Nginx reverse proxy.

    Quick Fix: The solution for me is to force-bind the container to the external interface, or even to all interfaces, e.g. -p 0.0.0.0:10022:22

    Real Fix (and root cause of problem): Plesk (server admin tool) added w/o prior notification an entry to /etc/docker/daemon.json to bind all container by default to 127.0.0.1 to prevent external access (1), which in the mean time got reverted back. However, every installation that had this update installed (thanks, auto-update...) still has this setting specified. You have to manually remove that entry from /etc/docker/daemon.json and restart the docker daemon.