I have a self-hosted Docker-Gitlab (v.16.8 currently) and it worked for some years. Then, a year ago I disabled it, recently enabled it again and updated it. The Gitlab webpage, the CI jobs, the Container Registry and all that works. What does not work, though, is my ability to interact with the server via Git through SSH, e.g. git clone ssh://git@INTERNETIP:10022/repo.git
Extended error log when connecting from my machine to the Docker-Gitlab instance hosted on an Ubuntu LTS (5.15.0-91) server
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/christian/.ssh/config line 1: Applying options for myurl.tld
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug2: resolving "myurl.tld" port 10022
debug3: resolve_host: lookup myurl.tld:10022
debug3: ssh_connect_direct: entering
debug1: Connecting to myurl.tld [externalip] port 10022.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: connect to address externalip port 10022: Connection refused
ssh: connect to host myurl.tld port 10022: Connection refused
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
The setup is the following:
22
, outside it is 10022
. Relevant entries of gitlab.rb
:external_url 'https://myurl.tld'
gitlab_rails['gitlab_shell_ssh_port'] = 10022
gitlab_sshd['enable'] = false # also tried with true (which is the new Gitlab-made SSH server)
#gitlab_sshd['listen_address'] = '[::]:22' # also tried 10022 here
docker run \
--detach \
--name gitlab \
--restart always \
-p 10022:22 -p 10080:80 -p 10443:443 -p 4567:4567 \
gitlab/gitlab-ce:latest
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
...
}
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "127.0.0.1",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
# nginx.conf
server {
list INTERNETIP:443 ssl http2;
server_name myurl.tld;
...
location ~ ^/.* {
proxy_pass http://0.0.0.0:10443;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
...
git clone ssh://git@INTERNETIP:10022/repo.git
but this fails the same way as just doing an SSH connect to this address (w/o the repo of course). The config file is set up and the SSH agent knows about the certificate.-A INPUT -p tcp -m tcp --dport 10022 -j ACCEPT
). I double verified for firewall issues by disabling the container and starting a Python-SimpleHTTPServer on that port 10022. I can access the webpage from my device.Public key for /etc/ssh/ssh_host_rsa_key does not match private key
debug1: private host key #0: ssh-rsa SHA256:xxx
Public key for /etc/ssh/ssh_host_ecdsa_key does not match private key
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:xxx
Public key for /etc/ssh/ssh_host_ed25519_key does not match private key
debug1: private host key #2: ssh-ed25519 SHA256:xxx
I have tracked the issue a bit:
git clone ssh://git@127.0.0.1:10022/repo.git
, a ssh -vvT git@172.17.0.3 -p 22
(the internal IP of the docker container), or even a ssh -vvvT git@127.0.0.1 -p 10022
, I get a successful connection and can issue SSH and GIT commands (also to my custom SSHD session from above).How can I further test/debug this matter and find out what stops me from connecting to the Git server as I did a year ago?
EDIT: Here are the current iptables rules, which were auto-generated by docker
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N f2b-ssh
-A INPUT -p tcp -m tcp --dport 22 -j f2b-ssh
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 4567 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A f2b-ssh -j RETURN
After some more inspection and tests, especially with the firewall, it dawned on me: it is the port mapping. Somewhen back, it seems the default behavior of docker to map ports has changed?!
My above script with -p 10022:22
maps the external port 10022 to the internal port 22, yet only on the interface 127.0.0.1 when I issue a docker ps
. This prevents me from accessing the system from the outside via port 10022. Then the question is, why does the webinterface on port 10443 work? Well, for this I had a reverse proxy entry set up, so the HTTP/HTTPS calls are actually coming from 127.0.0.1 due to the Nginx reverse proxy.
Quick Fix:
The solution for me is to force-bind the container to the external interface, or even to all interfaces, e.g. -p 0.0.0.0:10022:22
Real Fix (and root cause of problem):
Plesk (server admin tool) added w/o prior notification an entry to /etc/docker/daemon.json
to bind all container by default to 127.0.0.1
to prevent external access (1), which in the mean time got reverted back. However, every installation that had this update installed (thanks, auto-update...) still has this setting specified. You have to manually remove that entry from /etc/docker/daemon.json
and restart the docker daemon.