Calling Sftp with Expect fails inside a GitHub Workflow but works locally

I have a bash function that's called during a GitHub Workflow run. I'm using Sftp to change permissions on a bunch of remote files — without shell access Sftp is the only way available to me to change the permission using CHMOD.

When I run it in a Docker container with a specific version of Ubuntu and Expect — it runs perfectly. However, with the same OS environment and Expect — it fails in a GitHub Workflow.

The script `build-tool.sh`

ftp_put () {
    local   FILE_PERM=${1} \
            FILE_PATH=${2} \
            FTP_HOST=${3} \
            FTP_USER=${4} \
            FTP_PWD=${5}

    printf "\n🔒 Setting %s permissions for %s ...\n" "$FILE_PERM" "$FILE_PATH"

    # Connect to the FTP server over 
    expect << EOF 
        spawn sftp -v $FTP_USER@$FTP_HOST
        expect "*yes/no*" {
            set timeout 30
            send -- "yes\r";
            exp_continue
        }
        expect "*?assword:*" { 
            set timeout 30
            send -- "$FTP_PWD\r"
        }
        expect "Permission denied, please try again." {
            exit 1;
            exp_continue
        }        
        expect "sftp>*" { 
            send -- "cd $FILE_PATH\r"
        }
        expect "sftp>*" {
            send -- "chmod $FILE_PERM .\r"
        }
        expect "sftp>*" { 
            set timeout 60
            send -- "quit\n"
        }
        expect eof
EOF
}

...

Note: SFTP is being called with the verbose -v option, which gets passed to SSH.

Running locally (Docker container)

Locally, the script is called using remote FTP details which are stored in environment variables.

. ./build-tool.sh
ftp_put "$FILE_PERM" "$FILE_PATH" "$FTP_HOST" "$FTP_USER" "$FTP_PWD"

Output log

✅ The script successfully works, and applies the required file permissions:

🔒 Setting 555 permissions for app/sprinkles ...
spawn sftp -v mywebsite.com@ssh.mywebsite.com
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to ssh.mywebsite.com [46.30.211.80] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OneSSH-Proxy_1
debug1: compat_banner: no match: OneSSH-Proxy_1
debug1: Authenticating to ssh.mywebsite.com:22 as 'mywebsite.com'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:8sBBlU4Q4RqUzZO1J0RpAucj6/DicaJI1TjPDcRO22U
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ssh.mywebsite.com' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /root/.ssh/id_rsa
debug1: Will attempt key: /root/.ssh/id_ecdsa
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /root/.ssh/id_ed25519
debug1: Will attempt key: /root/.ssh/id_ed25519_sk
debug1: Will attempt key: /root/.ssh/id_xmss
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: password,publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/id_rsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ecdsa_sk
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Trying private key: /root/.ssh/id_ed25519_sk
debug1: Trying private key: /root/.ssh/id_xmss
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Next authentication method: password
mywebsite.com@ssh.mywebsite.com's password:
Authenticated to ssh.mywebsite.com ([46.30.211.80]:22) using "password".
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending environment.
debug1: Sending subsystem: sftp
debug1: Using server download size 261120
debug1: Using server upload size 261120
debug1: Server handle limit 1048571; using 64
Connected to ssh.mywebsite.com.
sftp> cd app/sprinkles
sftp> chmod 555 .
Changing mode on /customers/a/e/9/mywebsite.com/httpd.www/app/sprinkles/.
sftp> quit
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2516, received 2236 bytes, in 29.4 seconds
Bytes per second: sent 85.6, received 76.0
debug1: Exit status -1

Running in a GitHub Workflow

In GitHub, the workflow uses remote FTP server details which are stored in GitHub secrets. Other parts of the workflow use the values in these secrets — therefore the issue here isn't with the FTP password being incorrect.

Workflow step

    - name: "FTP: 🔏 Add write permissions"
      run: |
        . ./build-tool.sh
        ftp_put \
          755 \
          app/sprinkles \
          ${{ secrets.FTP_PROD_SERVER }} \
          ${{ secrets.FTP_PROD_USERNAME }} \
          ${{ secrets.FTP_PROD_PASSWORD }}

...

Output log

Note: I've tried putting the secrets in double quotes, and also to call ftp_put in a single line, without success.

❌ The script fails with:

spawn sftp -v ***@***
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to *** [***] port 22.
debug1: Connection established.
debug1: identity file /home/runner/.ssh/id_rsa type -1
debug1: identity file /home/runner/.ssh/id_rsa-cert type -1
debug1: identity file /home/runner/.ssh/id_ecdsa type -1
debug1: identity file /home/runner/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/runner/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/runner/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/runner/.ssh/id_ed25519 type -1
debug1: identity file /home/runner/.ssh/id_ed25519-cert type -1
debug1: identity file /home/runner/.ssh/id_ed25519_sk type -1
debug1: identity file /home/runner/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/runner/.ssh/id_xmss type -1
debug1: identity file /home/runner/.ssh/id_xmss-cert type -1
debug1: identity file /home/runner/.ssh/id_dsa type -1
debug1: identity file /home/runner/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OneSSH-Proxy_1
debug1: compat_banner: no match: OneSSH-Proxy_1
debug1: Authenticating to ***:22 as '***'
debug1: load_hostkeys: fopen /home/runner/.ssh/known_hosts: No such file or directory
debug1: load_hostkeys: fopen /home/runner/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:8sBBlU4Q4RqUzZO1J0RpAucj6/DicaJI1TjPDcRO22U
debug1: load_hostkeys: fopen /home/runner/.ssh/known_hosts: No such file or directory
debug1: load_hostkeys: fopen /home/runner/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: hostkeys_find_by_key_hostfile: hostkeys file /home/runner/.ssh/known_hosts does not exist
debug1: hostkeys_find_by_key_hostfile: hostkeys file /home/runner/.ssh/known_hosts2 does not exist
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts2 does not exist
The authenticity of host '*** (***)' can't be established.
ECDSA key fingerprint is SHA256:8sBBlU4Q4RqUzZO1J0RpAucj6/DicaJI1TjPDcRO22U.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
debug1: SELinux support disabled
Warning: Permanently added '***' (ECDSA) to the list of known hosts.
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/runner/.ssh/id_rsa 
debug1: Will attempt key: /home/runner/.ssh/id_ecdsa 
debug1: Will attempt key: /home/runner/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/runner/.ssh/id_ed25519 
debug1: Will attempt key: /home/runner/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/runner/.ssh/id_xmss 
debug1: Will attempt key: /home/runner/.ssh/id_dsa 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: password,publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/runner/.ssh/id_rsa
debug1: Trying private key: /home/runner/.ssh/id_ecdsa
debug1: Trying private key: /home/runner/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/runner/.ssh/id_ed25519
debug1: Trying private key: /home/runner/.ssh/id_ed25519_sk
debug1: Trying private key: /home/runner/.ssh/id_xmss
debug1: Trying private key: /home/runner/.ssh/id_dsa
debug1: Next authentication method: password
***@***'s password: 
debug1: Authentications that can continue: password,publickey
Permission denied, please try again.

What's going wrong here?

Solution

After echo'ing the GitHub secrets to the workflow output, with e.g. echo ${{ secrets.FTP_PROD_PASSWORD }} | sed 's/./& /g', I found that the FTP account that I'm using in the rest of the workflow is using regular non-encrypted FTP (Port 21), whereas the SFTP from my Docker container is using a SSH/SFTP login (Port 22). Problem solved! In hindsight, a simple mix-up.

Works perfectly.

🔒 Setting 755 permissions for app/sprinkles ...
spawn sftp ***@***

The authenticity of host '*** (***)' can't be established.
ECDSA key fingerprint is SHA256:**********/********.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '***' (ECDSA) to the list of known hosts.

***@***'s password: 
Connected to ***.
sftp> cd app/sprinkles
sftp> chmod 755 .
Changing mode on /customers/a/e/9/***/httpd.www/app/sprinkles/.
sftp> quit

🔒 Setting 755 permissions for app/vendor ...
spawn sftp ***@***

***@***'s password: 
Connected to ***.
sftp> cd app/vendor
sftp> chmod 755 .
Changing mode on /customers/a/e/9/***/httpd.www/app/vendor/.
sftp> quit

...

Calling Sftp with Expect fails inside a GitHub Workflow but works locally

The script build-tool.sh

Running locally (Docker container)

Output log

Running in a GitHub Workflow

Workflow step

Output log

The script `build-tool.sh`