Getting kex_exchange_identification: Connection closed by remote host from Github Action

I am trying to connect to a server behind cloudflared tunnel from Github Action in order rsync my project and it's giving a connection error. I've tried connecting through port 22 without the tunnel by allowing connection from the firewall and can confirm the ssh private key works. I believe it the issue is issue ssh configuration or cloudflare is blocking the connection because I do not see any failed login attempts from the remote server. How can I troubleshoot this?

---
name: test cloudflared

on:
  push:
    branches:
      - main
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  cloudflared:
    name: test cloudflared
    runs-on: ubuntu-latest
    timeout-minutes: 15

    defaults:
      run:
        shell: bash

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: install cloudflared
        run: |
          curl -L https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-archive-keyring.gpg >/dev/null
          echo "deb [signed-by=/usr/share/keyrings/cloudflare-archive-keyring.gpg] https://pkg.cloudflare.com/cloudflared $(lsb_release -cs) main" | sudo tee  /etc/apt/sources.list.d/cloudflared.list
          sudo apt update
          sudo apt-get install cloudflared

      - name: test cloudflared
        run: |
          which cloudflared
          whereis cloudflared
          /usr/local/bin/cloudflared --version

      - name: install-ssh-key
        uses: shimataro/ssh-key-action@v2
        with:
          key: ${{ secrets.SSH_PRIVATE_KEY }}
          name: id_rsa
          known_hosts: ${{ secrets.KNOWN_HOSTS }}
          config: |
            Host testhost
              HostName ${{ vars.HOST }}
              User ${{ vars.USER }}
              IdentityFile ~/.ssh/id_rsa
              ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h

      - name: test ssh
        run: ssh -vvv testhost 'whoami'

Github Action output:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.4, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/runner/.ssh/config
debug1: /home/runner/.ssh/config line 2: Applying options for testhost
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/runner/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/runner/.ssh/known_hosts2'
debug1: Executing proxy command: exec /usr/local/bin/cloudflared access ssh --hostname ssh.myhost.com
debug1: identity file /home/runner/.ssh/id_rsa type -1
debug1: identity file /home/runner/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.4
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535
Error: Process completed with exit code 255.

Solution

A couple things I had to do to get it working were:

Set up Service Token Auth in the ZT dashboard. That is, I had to create a Service Token ID and Secret pair under Access (and of course save the secret for later). Then, for the SSH application for the server, I needed to add a separate Service Auth Policy to its list of policies; in my case I only added the created Service Token under the list of includes.
Use said Service Token in the cloudflared access ssh ProxyCommand in my SSH config. That’s done by passing the client ID/secret you got from Cloudflare in the first step without the header names. So if you have CF-Access-Client-Id: abc and CF-Access-Client-Secret: xyz from the Cloudflare dashboard, you would pass in the arguments --id abc --secret xyz to the cloudflared access ssh ProxyCommand – in addition to the hostname.

Adding a couple more secrets to your repo to accommodate this, namely the aforementioned Service Token client/secret.

Here is an example of what it would look like:

name: Dispatch Jobs
on:
  push:
jobs:
  test:
    name: Deploy Server
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Install cloudflared
        run: |
          curl -L https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-archive-keyring.gpg >/dev/null
          echo "deb [signed-by=/usr/share/keyrings/cloudflare-archive-keyring.gpg] https://pkg.cloudflare.com/cloudflared $(lsb_release -cs) main" | sudo tee  /etc/apt/sources.list.d/cloudflared.list
          sudo apt update
          sudo apt install cloudflared
      - name: Configure SSH
        uses: shimataro/ssh-key-action@v2
        with:
          key: ${{ secrets.SSH_PRIVATE_KEY }}
          name: id_ed25519
          known_hosts: ${{ secrets.SSH_KNOWN_HOSTS }}
          config: |
            Host ${{ secrets.SSH_HOST }}
              ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h --id ${{ secrets.CF_SERVICE_TOKEN_ID }} --secret ${{ secrets.CF_SERVICE_TOKEN_SECRET }}
              IdentityFile ~/.ssh/id_ed25519
      # ... other build steps here ...
      - name: Copy Server Package to Server
        run: rsync -az --delete ./packages/server/server-*.tgz ${{ secrets.SSH_USER }}@${{ secrets.SSH_HOST }}:${{ secrets.SSH_TARGET_DIRECTORY }}