ruby-on-railssshcapistranoprivate-key

Capistrano asks for SSH password when it should use the SSH key

I have several Rails apps that use Capistrano.

When I sign in to the deployment server in the OSX Terminal.app I'm prompted for the password of my ida_rsa ssh key and I can successfully sign into the server with the given user.

Now when deploying to the same server with the same username I'm prompted for the server password of that user instead of the password to unlock my id_rsa private key.

I tried a number of things:

  1. Add ssh_options[:forward_agent] = true to deploy.rb
  2. ssh-copy-id this worked and I can already SSH to the server in the Terminal, which shows that this method does work.
  3. eval "$(ssh-agent -s)"

I would prefer to use my SSH key since I can't remember all of my servers passwords and I might want to disable password-SSH-login alltogether in the future too.

I know that my questions looks like an exact duplicate of Capistrano asks for password when deploying, despite SSH keys however, since none of the answers, including the approved answer did work for me I'm asking a new question.

My environment:

āžœ cat Gemfile.lock | grep capistrano | grep "("
    capistrano (3.17.1)
    capistrano-bundler (1.1.4)
      capistrano (~> 3.1)
    capistrano-puma (0.2.3)
      capistrano (>= 3.0)
    capistrano-rails (1.6.2)
      capistrano (~> 3.1)
      capistrano-bundler (>= 1.1, < 3)
    capistrano-rbenv (2.2.0)
      capistrano (~> 3.1)
  capistrano-bundler (~> 1.1.2)
  capistrano-rails (~> 1.1)

I'm using Rails 4.2.11.3

Permissions

I also tried to change the permissions of the files within ~/.ssh like so:

chmod u+rw,go-rwx .ssh/id_rsa
chmod u+rw,go-rwx .ssh/id_rsa.pub
chmod u+rw,go-rwx .ssh/authorized_keys
chmod u+rw,go-rwx .ssh/known_hosts

This did not result in any change regarding the password.

These are my ssh_optionsfrom my production.rb:

set :ssh_options, {
    forward_agent: true,
    verbose: :debug,
    auth_methods: %w(publickey password),
    keys: %w(~/.ssh/id_rsa)
}

Please note that the following in the terminal works fine:

ssh myproject@dev.example.com # id_rsa is used
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-57-generic x86_64)

Log

cap production deploy
      rbenv: rbenv_ruby is not set; ruby version will be defined by the remote hosts via rbenv
/usr/local/var/rbenv/versions/2.6.10/lib/ruby/2.6.0/open3.rb:213: warning: Insecure world writable dir /Users/me in PATH, mode 040777
00:00 git:push_sources
      01 git push origin main:main --force
      01 Everything up-to-date
    āœ” 01 me@localhost 0.528s
00:00 git:wrapper
      01 mkdir -p /tmp
D, [2023-03-31T12:31:54.063747 #8626] DEBUG -- net.ssh.transport.session[3fc260d9f0b4]: establishing connection to dev.example.com:22
D, [2023-03-31T12:31:54.084268 #8626] DEBUG -- net.ssh.transport.session[3fc260d9f0b4]: connection established
I, [2023-03-31T12:31:54.084647 #8626]  INFO -- net.ssh.transport.server_version[3fc260d9e614]: negotiating protocol version
D, [2023-03-31T12:31:54.084934 #8626] DEBUG -- net.ssh.transport.server_version[3fc260d9e614]: local is `SSH-2.0-Ruby/Net::SSH_4.2.0 x86_64-darwin20'
D, [2023-03-31T12:31:54.125834 #8626] DEBUG -- net.ssh.transport.server_version[3fc260d9e614]: remote is `SSH-2.0-OpenSSH_8.9p1 Ubuntu-3'
I, [2023-03-31T12:31:54.131316 #8626]  INFO -- net.ssh.transport.algorithms[3fc260da74a8]: sending KEXINIT
D, [2023-03-31T12:31:54.132134 #8626] DEBUG -- socket[3fc260d9ece0]: queueing packet nr 0 type 20 len 1156
D, [2023-03-31T12:31:54.132245 #8626] DEBUG -- socket[3fc260d9ece0]: sent 1160 bytes
D, [2023-03-31T12:31:54.143117 #8626] DEBUG -- socket[3fc260d9ece0]: read 1080 bytes
D, [2023-03-31T12:31:54.143224 #8626] DEBUG -- socket[3fc260d9ece0]: received packet nr 0 type 20 len 1076
I, [2023-03-31T12:31:54.143262 #8626]  INFO -- net.ssh.transport.algorithms[3fc260da74a8]: got KEXINIT from server
I, [2023-03-31T12:31:54.143331 #8626]  INFO -- net.ssh.transport.algorithms[3fc260da74a8]: negotiating algorithms
D, [2023-03-31T12:31:54.143454 #8626] DEBUG -- net.ssh.transport.algorithms[3fc260da74a8]: negotiated:
* kex: diffie-hellman-group-exchange-sha256
* host_key: ecdsa-sha2-nistp256
* encryption_server: aes128-ctr
* encryption_client: aes128-ctr
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client: 
* language_server: 
D, [2023-03-31T12:31:54.143481 #8626] DEBUG -- net.ssh.transport.algorithms[3fc260da74a8]: exchanging keys
D, [2023-03-31T12:31:54.143645 #8626] DEBUG -- socket[3fc260d9ece0]: queueing packet nr 1 type 34 len 20
D, [2023-03-31T12:31:54.143870 #8626] DEBUG -- socket[3fc260d9ece0]: sent 24 bytes
D, [2023-03-31T12:31:54.222189 #8626] DEBUG -- socket[3fc260d9ece0]: read 280 bytes
D, [2023-03-31T12:31:54.223277 #8626] DEBUG -- socket[3fc260d9ece0]: received packet nr 1 type 31 len 276
D, [2023-03-31T12:31:54.235760 #8626] DEBUG -- socket[3fc260d9ece0]: queueing packet nr 2 type 32 len 268
D, [2023-03-31T12:31:54.236019 #8626] DEBUG -- socket[3fc260d9ece0]: sent 272 bytes
D, [2023-03-31T12:31:54.254971 #8626] DEBUG -- socket[3fc260d9ece0]: read 504 bytes
D, [2023-03-31T12:31:54.255263 #8626] DEBUG -- socket[3fc260d9ece0]: received packet nr 2 type 33 len 484
D, [2023-03-31T12:31:54.257890 #8626] DEBUG -- socket[3fc260d9ece0]: queueing packet nr 3 type 21 len 20
D, [2023-03-31T12:31:54.257984 #8626] DEBUG -- socket[3fc260d9ece0]: sent 24 bytes
D, [2023-03-31T12:31:54.258046 #8626] DEBUG -- socket[3fc260d9ece0]: received packet nr 3 type 21 len 12
D, [2023-03-31T12:31:54.258392 #8626] DEBUG -- net.ssh.authentication.session[3fc260ce7e00]: beginning authentication of `myproject'
D, [2023-03-31T12:31:54.258557 #8626] DEBUG -- socket[3fc260d9ece0]: queueing packet nr 4 type 5 len 28
D, [2023-03-31T12:31:54.258615 #8626] DEBUG -- socket[3fc260d9ece0]: sent 52 bytes
D, [2023-03-31T12:31:54.335033 #8626] DEBUG -- socket[3fc260d9ece0]: read 52 bytes
D, [2023-03-31T12:31:54.335202 #8626] DEBUG -- socket[3fc260d9ece0]: received packet nr 4 type 6 len 28
D, [2023-03-31T12:31:54.335309 #8626] DEBUG -- net.ssh.authentication.session[3fc260ce7e00]: trying publickey
D, [2023-03-31T12:31:54.337096 #8626] DEBUG -- net.ssh.authentication.agent[3fc260cf75bc]: connecting to ssh-agent
D, [2023-03-31T12:31:54.337260 #8626] DEBUG -- net.ssh.authentication.agent[3fc260cf75bc]: sending agent request 1 len 47
D, [2023-03-31T12:31:54.337406 #8626] DEBUG -- net.ssh.authentication.agent[3fc260cf75bc]: received agent packet 5 len 1
D, [2023-03-31T12:31:54.337444 #8626] DEBUG -- net.ssh.authentication.agent[3fc260cf75bc]: sending agent request 11 len 0
D, [2023-03-31T12:31:54.337553 #8626] DEBUG -- net.ssh.authentication.agent[3fc260cf75bc]: received agent packet 12 len 313
D, [2023-03-31T12:31:54.337775 #8626] DEBUG -- net.ssh.authentication.methods.publickey[3fc260cf3868]: trying publickey (9f:9e:f5:2a:11:d2:c2:53:fb:17:44:81:b5:65:16:7c)
D, [2023-03-31T12:31:54.337956 #8626] DEBUG -- socket[3fc260d9ece0]: queueing packet nr 5 type 50 len 348
D, [2023-03-31T12:31:54.338032 #8626] DEBUG -- socket[3fc260d9ece0]: sent 372 bytes
D, [2023-03-31T12:31:54.361596 #8626] DEBUG -- socket[3fc260d9ece0]: read 68 bytes
D, [2023-03-31T12:31:54.361844 #8626] DEBUG -- socket[3fc260d9ece0]: received packet nr 5 type 51 len 44
D, [2023-03-31T12:31:54.361938 #8626] DEBUG -- net.ssh.authentication.session[3fc260ce7e00]: allowed methods: publickey,password
E, [2023-03-31T12:31:54.361985 #8626] ERROR -- net.ssh.authentication.session[3fc260ce7e00]: all authorization methods failed (tried publickey)
#<Thread:0x00007f84c1a7d148@/usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
/usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as myproject@dev.example.com: Authentication failed for user myproject@dev.example.com (SSHKit::Runner::ExecuteError)
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/net-ssh-4.2.0/lib/net/ssh.rb:254:in `start': Authentication failed for user myproject@dev.example.com (Net::SSH::AuthenticationFailed)
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in `call'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in `with'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:177:in `with_ssh'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:130:in `execute_command'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `tap'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:80:in `execute'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/capistrano-3.17.1/lib/capistrano/scm/tasks/git.rake:8:in `block (3 levels) in eval_rakefile'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in `run'
        from /usr/local/var/rbenv/versions/2.6.10/lib/ruby/gems/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as myproject@dev.example.com: Authentication failed for user myproject@dev.example.com


Caused by:
Net::SSH::AuthenticationFailed: Authentication failed for user myproject@dev.example.com

Tasks: TOP => deploy:check => git:check => git:wrapper
(See full trace by running task with --trace)
The deploy has failed with an error: Exception while executing as myproject@dev.example.com: Authentication failed for user myproject@dev.example.com


** DEPLOY FAILED
** Refer to log/capistrano.log for details. Here are the last 20 lines:


  INFO START 2023-03-31 12:28:44 +0200 cap production deploy

  INFO ---------------------------------------------------------------------------

  INFO rbenv: rbenv_ruby is not set; ruby version will be defined by the remote hosts via rbenv

  INFO [952e12b2] Running /usr/bin/env git push origin main:main --force as me@localhost

  INFO [952e12b2] Finished in 0.580 seconds with exit status 0 (successful).

  INFO [6a82481f] Running /usr/bin/env mkdir -p /tmp as myproject@dev.example.com

  INFO ---------------------------------------------------------------------------

  INFO START 2023-03-31 12:29:39 +0200 cap production deploy

  INFO ---------------------------------------------------------------------------

  INFO rbenv: rbenv_ruby is not set; ruby version will be defined by the remote hosts via rbenv

  INFO [eba965ea] Running /usr/bin/env git push origin main:main --force as me@localhost

  INFO [eba965ea] Finished in 0.604 seconds with exit status 0 (successful).

  INFO [9ec098a4] Running /usr/bin/env mkdir -p /tmp as myproject@dev.example.com

  INFO ---------------------------------------------------------------------------

  INFO START 2023-03-31 12:31:53 +0200 cap production deploy

  INFO ---------------------------------------------------------------------------

  INFO rbenv: rbenv_ruby is not set; ruby version will be defined by the remote hosts via rbenv

  INFO [b15f7565] Running /usr/bin/env git push origin main:main --force as me@localhost

  INFO [b15f7565] Finished in 0.528 seconds with exit status 0 (successful).

  INFO [b3ed8356] Running /usr/bin/env mkdir -p /tmp as myproject@dev.example.com
Solution

  • I'm not really sure why updating net-ssh gem works. But I did a bit of debugging:

    # net-ssh v6.1.0
>> Net::SSH.start("myhost.com", "root", keys: ["/home/alex/.ssh/id_rsa"])
/home/alex/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/net-ssh-6.1.0/lib/net/ssh.rb:268:in `start':
Authentication failed for user root@myhost.com (Net::SSH::AuthenticationFailed)

    On the server:

    $ tail -f /var/log/auth.log
Apr  3 04:15:42 myhost sshd[2548315]: userauth_pubkey: 
  key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
#          ^              ^

    Add ssh-rsa to accepted algorithms

    not recommended

    # /etc/ssh/sshd_config

PubKeyAcceptedAlgorithms +ssh-rsa

    sudo service ssh restart

    Verify:

    >> Net::SSH.start("myhost.com", "root", keys: ["/home/alex/.ssh/id_rsa"]) do |ssh|
     ssh.exec!("hostname")
   end
=> "myhost\n"

    Update net-ssh

    Updating to net-ssh version 7 also fixes the issue. Not sure which config changed, but I'm thinking maybe this:

    https://github.com/net-ssh/net-ssh/commit/a45f54fe1de434605af0b7195dd9a91bccd2cec5

    update

    I've checked that ^ commit, that's the one. I've also noticed this commit on mrsk that sums it up:

    Require net-ssh ~> 7.0 for SHA-2 support
    Versions of net-ssh before 7.0 do not support the SHA-2 algorithm and result in mrsk not being able to connect to hosts using keys generated with it. net-ssh is also a dependency of sshkit, however, sshkit has a version requirement of >= 2.8.0 for net-ssh, so is not effective at ensuring mrsk has the version it needs to be the most compatible.