OpenSSL Nonblocking Socket Accept And Connect Failed

Here is my question: Is it bad to set socket to nonblocking before I call accept or connect? or it should be using blocking accept and connect, then change the socket to nonblocking?

I'm new to OpenSSL and not very experienced with network programming. My problem here is I'm trying to use OpenSSL with a nonblocking socket network to add security. When I call SSL_accept on server side and SSL_connect on client side, and check return error code using

SSL_get_error(m_ssl, n);
char error[65535];
ERR_error_string_n(ERR_get_error(), error, 65535);

the return code from SSL_get_error indicates SSL_ERROR_WANT_READ, while ERR_error_string_n prints out "error:00000000:lib(0):func(0):reason(0)", which I think it means no error. SSL_ERROR_WANT_READ means I need to retry both SSL_accept and SSL_connect.

Then I use a loop to retry those function, but this just leads to a infinite loop :(

I believe I have initialized SSL properly, here is the code

//CRYPTO_malloc_init(); 
SSL_library_init();

const SSL_METHOD *method;

// load & register all cryptos, etc.
OpenSSL_add_all_algorithms();
// load all error messages
SSL_load_error_strings();

if (server) {
    // create new server-method instance
    method = SSLv23_server_method();
}
else {
    // create new client-method instance
    method = SSLv23_client_method();
}

// create new context from method
m_ctx = SSL_CTX_new(method);
if (m_ctx == NULL) {
    throwError(-1);
}

If there is any part I haven't mentioned but you think it could be the problem, please let me know.

SSL_ERROR_WANT_READ means I need to retry both SSL_accept and SSL_connect.

Yes, but this is not the full story. You should retry the call only after the socket gets readable, e.g. you need to use select or poll or similar functions to wait, until the socket gets readable. Same applies to SSL_ERROR_WANT_WRITE, but here you have to wait for the socket to get writable.

If you just retry without waiting it will probably eventually succeed, but only after having 100s of failed calls. While doing select does not guarantee that it will succeed immediately at the next call it will take only a few calls of SSL_connect/SSL_accept until it succeeds and it will not busy loop and eat CPU in the mean time.