csslopenssllibssl

How to handle SSL_ERROR_SSL on SSL_accept


I'm using OpenSSL (the development library, <openssl/ssl.h>) to program a small, simple server in C.

Since I'm only testing it locally, I use a self-signed certificate. When first connecting, since my browser doesn't trust the certificate, SSL_accept() returns SSL_ERROR_SSL, which is fine, that's what should happen.

I'm wondering how I should handle the SSL_ERROR_SSL error, since the docs on SSL_get_error() say:

SSL_ERROR_SSL
A non-recoverable, fatal error in the SSL library occurred, usually a protocol error. The OpenSSL error queue contains more information on the error. If this error occurs then no further I/O operations should be performed on the connection and SSL_shutdown() must not be called.

If I can't can SSL_shutdown(), what should I do with the SSL connection? How do I reuse the SSL* object? And why can't I call SSL_shutdown()?

I tried looking at the error queue, which gave me the expected error of untrusted certificate. The OpenSSL site doesn't have any answers, and it seems that most programs try to avoid handling this problem instead of dealing with it.


Solution

  • If I can't can SSL_shutdown(), what should I do with the SSL connection? How do I reuse the SSL* object? And why can't I call SSL_shutdown()?

    You can't use SSL_shutdown() in this case because it is a non-recoverable, fatal error. The SSL session failed to be established, and the state of the socket is unknown. So there is nothing you can reliably shut down. And, the SSL object cannot be reused.

    So, the only thing you can do is destroy the SSL object, close the socket, and move on.