I am using libssh2, where my flow ends up calling _libssh2_userauth_publickey, which itself calls _libssh2_ntohu32(pubkeydata). My understanding is that it should return a small number (e.g. 7 for "ssh-rsa", but in my case it returns garbage (e.g. 813826570).
My guess is that the pubkeydata I am giving is in the wrong format (it should start with something like 00 00 00 07 for "ssh-rsa", and obviously it doesn't).
The bytes I am passing as pubkeydata are my id_rsa.pub converted to PEM and then to bytes, using e.g.:
ssh-keygen -e -m pem -f id_rsa.pub | openssl asn1parse -dump -out pubkeydata.bin
But this doesn't start with 00 00 00 07, which makes me believe this is not what _libssh2_userauth_publickey expects.
What format does _libssh2_userauth_publickey expect, and how do I convert my id_rsa.pub to it?
It seems like pubkeydata is "simply" the decoded value of the base64-encoded part of the .pub file.
Say id_rsa.pub looks like this:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQClFuoMbXDHrs0EWyMvWueEGz9pJjsszh8bcPGFzSmux2bFmM3G0nCTSRc+svifaTPMO+WB865tPpkpEWNNUsKPmzlKHy1zPRo5FJMmfulExJCQGKPF6B9rqJHnjILsVvAAtarDyPag/ [...] PBcEhS8YA7xn60zhXzV+0F6i6DVo6hN2TfoK85zGG59cf1lXwrUjS4hXu1XjjDavZOBBCk1Q5aIA+cVuUJWg3NE9mQYRpPSyn9L+rs+SpoS2YX/oRBD2wnlu7czU9jY6NwCedIeJI+D/aLDLqW9GNp2B4XVjvrMWa+Q==
The base64-encoded value is AAAAB3Nza...a+Q==. Decode this and pass it as pubkeydata.