libssh: Public key authentication failed: The key algorithm 'ssh-rsa' is not allowed to be used by PUBLICKEY_ACCEPTED_TYPES configuration option

I'm using libssh to connect to a remote host running debian 8.11 and OpenSSH_6.7p1. The host has my RSA public key.

I can connect to the remote host via the command line. However, using libssh to connect to the virtual machine it fails giving the following error:

"Public key authentication failed: The key algorithm 'ssh-rsa' is not allowed to be used by PUBLICKEY_ACCEPTED_TYPES configuration option"

This is where it is failing:

static ssh_session start_session(const char* host, const char* user, const char* keyfile, const char* port) {
ssh_session session = ssh_new();
if (session == NULL) {
    fprintf(stderr, "Error creating SSH session\n");
    exit(EXIT_FAILURE);
}

ssh_options_set(session, SSH_OPTIONS_HOST, host);
ssh_options_set(session, SSH_OPTIONS_USER, user);
ssh_options_set(session, SSH_OPTIONS_PORT_STR, port);

int rc = ssh_connect(session);
if (rc != SSH_OK) {
    fprintf(stderr, "Error connecting to virtual machine: %s\n", ssh_get_error(session));
    ssh_free(session);
    exit(EXIT_FAILURE);
}

rc = auth_keyfile(session, keyfile);
if (rc != SSH_AUTH_SUCCESS) {
    fprintf(stderr, "Error authenticating with virtual machine\n");
    ssh_disconnect(session);
    ssh_free(session);
    exit(EXIT_FAILURE);
}

return session;

}

I already tried updating the ssh version, but it still failed. What might be causing this problem?

Solution

The ssh-rsa algorithm is kind of deprecated nowadays. You can still use the keys, but the protocol that is used on will be rsa-sha2-256 or rsa-sha2-512, instead.

From the error message it appears that libssh does not enable ssh-rsa as the key exchange algorithm anymore by default (which in principle is a good thing). You can explicitly reenable it by setting the SSH_OPTIONS_PUBLICKEY_ACCEPTED_TYPES ssh option via ssh_options_set() to a string that includes ssh-rsa.

For example:

ssh_options_set(session, SSH_OPTIONS_PUBLICKEY_ACCEPTED_TYPES, "rsa-sha2-256,rsa-sha2-512,ecdh-sha2-nistp256,ssh-rsa");

(This list still contains other sensible algorithms in case you are talking to other SSH servers also, because newer SSH servers will actually reject connections if you only have ssh-rsa support active without supporting the newer algorithms.)

ssh-rsa is deprecated for a reason though, because it uses the insecure SHA-1 hash algorithm. If you need to connect to an old system that runs an OpenSSL version before 7.2 (when the rsa-sha2-* algorithms were introduced), then ssh-rsa is the only way you can do public key authentication against such a system, so you kind of have not much of a choice.

Ideally though you'd upgrade your Debian box to a Debian version that is still supported, because newer Debian versions come with newer OpenSSH versions, and those do support the newer algorithms.