Seems like most people that are having this problem are experiencing the issue on the remote server (i.e. running Git on the remote). I am having this problem at the local level (no AgentForwarding involved).
When I attempt to connect to my server with SSH-Remote, I am immediately prompted with an "Enter your passphrase" text box at the top (followed by a second immediately after I connect to the server). If I open my local terminal:
Load the key/passphrase by running a ssh user@remote, entering passphrase, connecting and then exiting.
I can confirm ssh-agent is running via ps -e | grep ssh.
I can use ssh-agent locally--if I ssh user@remote again, I am NOT prompted for a passphrase, which is the expected behavior.
After all this confirmation, if I open up VSCode and run SSH-Remote to connect to server, I am again immediately prompted for a passphrase (again, twice--once before connection and once after).
I have no extensions installed, this isn't for Git, nor am I trying to use my ssh-agent on the remote. I just don't want to be asked for a passphrase multiple times every single time I open the connection, open a new folder, etc...
Any advice is appreciated. Thanks!
I am using zsh as my default shell (its also set as the default shell in VSCode) and the ssh-agent zsh plugin to start ssh-agent and automatically add keys.
This is the content of my ~/.ssh/config:
AddKeysToAgent yes
Host server
HostName 192.168.1.155
User user
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
Host github.com
HostName github.com
IdentityFile ~/.ssh/id_ed25519_git
IdentitiesOnly yes
I tried to get as much logging as possible from VSCode's SSH startup process, and I couldn't even find a reference to the use of ssh-agent or $SSH_AUTH_SOCK.
[22:26:29.832] stderr> debug1: Found key in /home/user/.ssh/known_hosts:1
[22:26:29.834] stderr> debug1: rekey out after 134217728 blocks
[22:26:29.834] stderr> debug1: SSH2_MSG_NEWKEYS sent
[22:26:29.834] stderr> debug1: expecting SSH2_MSG_NEWKEYS
[22:26:29.834] stderr> debug1: SSH2_MSG_NEWKEYS received
[22:26:29.835] stderr> debug1: rekey in after 134217728 blocks
[22:26:29.835] stderr> debug1: Will attempt key: /home/user/.ssh/id_ed25519 ED25519 XXXXXXXXXXXXXXXXXXXXXXXXXX explicit
[22:26:29.835] stderr> debug1: SSH2_MSG_EXT_INFO received
[22:26:29.835] stderr> debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
[22:26:29.835] stderr> debug1: kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0>
[22:26:29.880] stderr> debug1: SSH2_MSG_SERVICE_ACCEPT received
[22:26:29.892] stderr> debug1: Authentications that can continue: publickey
[22:26:29.892] stderr> debug1: Next authentication method: publickey
[22:26:29.892] stderr> debug1: Offering public key: /home/user/.ssh/id_ed25519 ED25519 XXXXXXXXXXXXXXXXXXXXXXXXXX explicit
[22:26:29.893] stderr> debug1: Server accepts key: /home/user/.ssh/id_ed25519 ED25519 XXXXXXXXXXXXXXXXXXXXXXXXXX explicit
[22:26:29.893] stderr> debug1: read_passphrase: can't open /dev/tty: No such device or address
[22:26:29.956] Got askpass request: {"request":"Enter passphrase for key '/home/user/.ssh/id_ed25519':"}
[22:26:29.956] Detected passphrase message
[22:26:29.957] Listening for interwindow password on /run/user/1000/vscode-ssh-askpass-a712a2dfc9c305b1ae74bbab98be91ebec02eafd.sock
[22:26:29.957] Writing password prompt to globalState
[22:26:31.870] Got passphrase response
[22:26:31.870] Interactor gave response: ********************
[22:26:31.870] Cleaning up other-window auth server
[22:26:32.000] stderr> Authenticated to 192.168.1.120 ([192.168.1.120]:22) using "publickey".
Finally managed to figure out a way to force -vvv. They do not make it easy for absolutely 0 reason whatsoever. In this block, I found one mention of agent and it was a log entry saying there isn't one. Not sure how it decides that (which environment variable).
[13:41:11.503] > local-server-1> Running ssh connection command: "-vvv -T -D 33699 -o ConnectTimeout=15 -F /home/user/.ssh/config server bash"
[13:41:11.505] > local-server-1> Spawned ssh, pid=51235
[13:41:11.508] stderr> OpenSSH_9.5p1, OpenSSL 3.1.4 24 Oct 2023
[13:41:11.508] stderr> debug1: Reading configuration data /home/user/.ssh/config
[13:41:11.508] stderr> debug1: /home/user/.ssh/config line 3: Applying options for *
[13:41:11.508] stderr> debug1: /home/user/.ssh/config line 6: Applying options for server
[13:41:11.508] stderr> debug2: resolve_canonicalize: hostname 192.168.1.120 is address
[13:41:11.508] stderr> debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
[13:41:11.508] stderr> debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
[13:41:11.508] stderr> debug3: ssh_connect_direct: entering
[13:41:11.508] stderr> debug1: Connecting to 192.168.1.120 [192.168.1.120] port 22.
[13:41:11.508] stderr> debug3: set_sock_tos: set socket 3 IP_TOS 0x48
[13:41:11.508] stderr> debug2: fd 3 setting O_NONBLOCK
[13:41:11.508] stderr> debug1: fd 3 clearing O_NONBLOCK
[13:41:11.508] stderr> debug1: Connection established.
[13:41:11.508] stderr> debug3: timeout: 15000 ms remain after connect
[13:41:11.508] stderr> debug1: identity file /home/user/.ssh/id_ed25519 type 3
[13:41:11.508] stderr> debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
[13:41:11.508] stderr> debug1: Local version string SSH-2.0-OpenSSH_9.5
[13:41:11.519] stderr> debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u1
[13:41:11.519] stderr> debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u1 pat OpenSSH* compat 0x04000000
[13:41:11.519] stderr> debug2: fd 3 setting O_NONBLOCK
[13:41:11.519] stderr> debug1: Authenticating to 192.168.1.120:22 as 'ssh-user'
[13:41:11.519] stderr> debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:1
[13:41:11.519] stderr> debug3: record_hostkey: found key type RSA in file /home/user/.ssh/known_hosts:2
[13:41:11.519] stderr> debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:3
[13:41:11.519] stderr> debug3: load_hostkeys_file: loaded 3 keys from 192.168.1.120
[13:41:11.519] stderr> debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
[13:41:11.519] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
[13:41:11.519] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
[13:41:11.519] stderr> debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
[13:41:11.519] stderr> debug3: send packet: type 20
[13:41:11.519] stderr> debug1: SSH2_MSG_KEXINIT sent
[13:41:11.520] stderr> debug3: receive packet: type 20
[13:41:11.520] stderr> debug1: SSH2_MSG_KEXINIT received
[13:41:11.520] stderr> debug2: local client KEXINIT proposal
[13:41:11.520] stderr> debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
[13:41:11.520] stderr> debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
[13:41:11.520] stderr> debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
[13:41:11.520] stderr> debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
[13:41:11.520] stderr> debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[13:41:11.520] stderr> debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
[13:41:11.520] stderr> debug2: compression ctos: none,zlib@openssh.com,zlib
[13:41:11.520] stderr> debug2: compression stoc: none,zlib@openssh.com,zlib
[13:41:11.520] stderr> debug2: languages ctos:
[13:41:11.520] stderr> debug2: languages stoc:
[13:41:11.520] stderr> debug2: first_kex_follows 0
[13:41:11.520] stderr> debug2: reserved 0
[13:41:11.520] stderr> debug2: peer server KEXINIT proposal
[13:41:11.520] stderr> debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
[13:41:11.520] stderr> debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
[13:41:11.520] stderr> debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
[13:41:11.520] stderr> debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
[13:41:11.520] stderr> debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
[13:41:11.520] stderr> debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
[13:41:11.520] stderr> debug2: compression ctos: none
[13:41:11.520] stderr> debug2: compression stoc: none
[13:41:11.521] stderr> debug2: languages ctos:
[13:41:11.521] stderr> debug2: languages stoc:
[13:41:11.521] stderr> debug2: first_kex_follows 0
[13:41:11.521] stderr> debug2: reserved 0
[13:41:11.521] stderr> debug1: kex: algorithm: curve25519-sha256@libssh.org
[13:41:11.521] stderr> debug1: kex: host key algorithm: ssh-ed25519
[13:41:11.521] stderr> debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
[13:41:11.521] stderr> debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
[13:41:11.521] stderr> debug3: send packet: type 30
[13:41:11.521] stderr> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
[13:41:11.526] stderr> debug3: receive packet: type 31
[13:41:11.526] stderr> debug1: SSH2_MSG_KEX_ECDH_REPLY received
[13:41:11.526] stderr> debug1: Server host key: ssh-ed25519 SHA256:+wYnNtg/GzeBjkCVw2VW+1KuZHq1808nstVdkZh1+bU
[13:41:11.526] stderr> debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:1
[13:41:11.526] stderr> debug3: record_hostkey: found key type RSA in file /home/user/.ssh/known_hosts:2
[13:41:11.526] stderr> debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:3
[13:41:11.526] stderr> debug3: load_hostkeys_file: loaded 3 keys from 192.168.1.120
[13:41:11.526] stderr> debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
[13:41:11.526] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
[13:41:11.526] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
[13:41:11.526] stderr> debug1: Host '192.168.1.120' is known and matches the ED25519 host key.
[13:41:11.526] stderr> debug1: Found key in /home/user/.ssh/known_hosts:1
[13:41:11.528] stderr> debug3: send packet: type 21
[13:41:11.528] stderr> debug2: ssh_set_newkeys: mode 1
[13:41:11.528] stderr> debug1: rekey out after 134217728 blocks
[13:41:11.528] stderr> debug1: SSH2_MSG_NEWKEYS sent
[13:41:11.528] stderr> debug1: expecting SSH2_MSG_NEWKEYS
[13:41:11.528] stderr> debug3: receive packet: type 21
[13:41:11.528] stderr> debug1: SSH2_MSG_NEWKEYS received
[13:41:11.528] stderr> debug2: ssh_set_newkeys: mode 0
[13:41:11.528] stderr> debug1: rekey in after 134217728 blocks
[13:41:11.528] stderr> debug1: Will attempt key: /home/user/.ssh/id_ed25519 ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX explicit
[13:41:11.528] stderr> debug2: pubkey_prepare: done
[13:41:11.528] stderr> debug3: send packet: type 5
[13:41:11.528] stderr> debug3: receive packet: type 7
[13:41:11.528] stderr> debug1: SSH2_MSG_EXT_INFO received
[13:41:11.528] stderr> debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
[13:41:11.528] stderr> debug1: kex_ext_info_check_ver: publickey-hostbound@openssh.com=<0>
[13:41:11.575] stderr> debug3: receive packet: type 6
[13:41:11.576] stderr> debug2: service_accept: ssh-userauth
[13:41:11.576] stderr> debug1: SSH2_MSG_SERVICE_ACCEPT received
[13:41:11.576] stderr> debug3: send packet: type 50
[13:41:11.599] stderr> debug3: receive packet: type 51
[13:41:11.599] stderr> debug1: Authentications that can continue: publickey
[13:41:11.599] stderr> debug3: start over, passed a different list publickey
[13:41:11.599] stderr> debug3: preferred publickey,keyboard-interactive,password
[13:41:11.599] stderr> debug3: authmethod_lookup publickey
[13:41:11.599] stderr> debug3: remaining preferred: keyboard-interactive,password
[13:41:11.599] stderr> debug3: authmethod_is_enabled publickey
[13:41:11.599] stderr> debug1: Next authentication method: publickey
[13:41:11.599] stderr> debug1: Offering public key: /home/user/.ssh/id_ed25519 ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX explicit
[13:41:11.599] stderr> debug3: send packet: type 50
[13:41:11.599] stderr> debug2: we sent a publickey packet, wait for reply
[13:41:11.599] stderr> debug3: receive packet: type 60
[13:41:11.599] stderr> debug1: Server accepts key: /home/user/.ssh/id_ed25519 ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX explicit
[13:41:11.600] stderr> debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX
[13:41:11.600] stderr> debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX
[13:41:11.600] stderr> debug1: read_passphrase: can't open /dev/tty: No such device or address
[13:41:11.663] Got askpass request: {"request":"Enter passphrase for key '/home/user/.ssh/id_ed25519':"}
[13:41:11.663] Detected passphrase message
[13:41:11.663] Listening for interwindow password on /run/user/1000/vscode-ssh-askpass-3c669b661bf05bd38516123cc0c4977123a4eed3.sock
[13:41:11.663] Writing password prompt to globalState
[13:41:23.598] Got passphrase response
[13:41:23.599] Interactor gave response: ********************
[13:41:23.599] Cleaning up other-window auth server
[13:41:23.724] stderr> debug3: no authentication agent, not adding key
[13:41:23.724] stderr> debug3: send packet: type 50
[13:41:23.731] stderr> debug3: receive packet: type 52
[13:41:23.731] stderr> Authenticated to 192.168.1.120 ([192.168.1.120]:22) using "publickey".
At this point, the only thing I can imagine, given that the local-server can't find the agent, is that (despite setting the shell to ZSH in VSCode settings), Remote SSH is still using bash or sh, and because of that, has no access to the environment variables.
Not sure how I can tell if this is the case. Even more unsure of how I would change that.
Anywhere else I should look for logs to diagnose the root cause of this issue?
Probably will be my last update, since I found a somewhat workable solution, but I would like to still understand why this is giving me trouble and fix it if possible.
If I make sure that no other VSCode instances are running, and run code from the command line, that specific VSCode instance can interface with the ssh-agent and doesn't ask for passphrases. If I run it from KRunner (on Arch Linux KDE), it can not. Not sure if this is a permissions or environment variable thing at this point.
Given this new information, any ideas on next step troubleshooting? Thanks again.
Resolved. I was using the oh-my-zsh ssh-agent plugin which initialized and exported the ssh-agent environment variables for individual interactive shell instances on shell startup, since it was sourced in .zshrc. VSCode can't access that if it is run via xdg-open/kde-open. Running it in the console actually allowed it to work which was the hint I needed to resolve the issue.
The important lesson is understanding the differences between interactive, non-interactive, and login shells and how environment variables may or may not be set in one context or another. Depending on the functionality you are aiming for, you should be using one of (not exhaustive): .zshrc, .zprofile, OR .bashrc, .bash_profile. I recommend you initialize your ssh-agent in a profile file for your respective shell IF you plan to use ssh from beyond the shell. That way, even those programs can access the necessary env variables.
It was somewhat disappointing. Heres the solution plus reasoning and sources in case others run into this:
code open and opened it via the console (code) instead of the code.desktop, it actually worked with the ssh-agent! I had tried this before, but I suspect it was done incorrectly for two reasons:code open, and seemingly even if I run code from the terminal, if I had a running instance started from the .desktop, it wouldn't work. My guess is that consecutive calls to code trigger the New Window functionality instead of allowing another instance completely. This makes sense as under the process tree, there remains only one parent code instance.echo $SSH_AUTH_SOCK yielded correct results from INSIDE the VSCode terminal) then the entire instance of VSCode also had access to those variables. This is not true. I got duped by this and overlooked the issue of environment variables after trying it, hence why this took me so long.This meant it must have been an environment variable issue. To confirm, I added to the .desktop the current SSH_AUTH_SOCK and SSH_AGENT_PID env variables. Lo and behold, it worked.
Quick Google search regarding why the terminal might have access to env variables and not xdg-open (or possibly kde-open in my case) yielded this thread. In my case, I am using the oh-my-zsh ssh-agent plugin that is started in .zshrc--which is only run for interactive terminals, on each terminal instance startup. My guess was that xdg-open/kde-open are not interactive terminals, as they only run the Exec portion of the .desktop with no user input. Thus, the plugin wasn't starting and the variables were not being exported for the VSCode startup. Rather, if I run code at the terminal, since it was born from that terminal, it inherits the environment variables the terminal initialized which is why it was working in that context. Ugh.
More research pointed to moving ssh-agent initialization out to .zprofile. So, I moved the ssh-agent initialization out to $HOME/.zprofile and made an equivalent initialization (checks if ssh-agent is running first before so they don't each start their own instance) in $HOME/.bash_profile.
It worked at last.
This was the result of ~4 hours of debugging/researching each day for the last 4 or 5 days, so kinda pathetic it was a relatively simple issue lol.