I am writing a client/server TCP program and wish to encrypt and authenticate the connection using TLS. My plan was for clients to submit public RSA keys to the server and then use the private key and a username both to authenticate the client's identity and encrypt the connection.
The GnuTLS documentation leaves a lot to be desired, but I found a good PSK example and tried to adapt it to use RSA asymmetrical keys. After banging my head against this problem, in particular all sorts of obscure subtleties which are not clearly described by the documentation, I have got the client to stop complaining about "No supported cipher suites have been found" but now the server does.
After yet more searching of the internet I found a test program that is part of GnuTLS CI that tests RSA-PSK and I realised that the shared key is still symmetrical. RSA-PSK is not what I thought it was.
I really don't like using a symmetrical key as it needs to be stored in plain on the server side and is thus insecure.
However a brief look at RSA key exchange suggests that there is no username involved and it is actually certificate based, and that the server uses its own private key while all clients use the same public key. Thus it is impossible to guarantee a client's identity.
After spending a day trying to get TLS to work, and being loathe to re-invent the wheel (which is always a security risk) I am asking here in case there are any experts who can advise me on how to achieve what I am seeking to achieve.
The only solution that springs to mind is rather dirty and hacky (and reinventing the wheel), namely to implement a pre-TLS-handshake handshake where the client generates a random key, encrypts it with its private key, and sends it and the username to the server which then uses the public key to decrypt it. GnuTLS can then use normal symmetric PSK key exchange using the random key.
I don't understand why GnuTLS doesn't implement something like this anyway. What I was expecting was for the GnuTLS client to send the username to the server, the server then performs key exchange by generating keys for the encryption and encrypts them using the client's public key and sends the encrypted key to the client. The client decrypts the key using its private key and lo and behold, key exchange is securely complete and the TLS channel is open.
So in summary:
1) How do I achieve what I need to achieve (preferably using GnuTLS)?
2) Why doesn't RSA-PSK work as I expect and instead use symmetric keys and a certificate?
TLS is unsuitable for the purpose to which I was trying to put it. Libssh would probably be the best solution but I have already finished a full solution using libgcrypt, so...