I am trying to use BoringSSL in a client that talks to the server which uses OpenSSL. Both the client and server are sitting in internal network and communicate with private ip address (192.168.x.x). One of the two functions exposed by BoringSSL library to create a SSL connection with SSL_CTX_new() is TLS_with_buffers_method(). I have tried this function and it failed to establish with my OpenSSL server because the certificate from server is self-signed.
According to BoringSSL's porting guide below, it avoids creating X509 objects.
"The function TLS_with_buffers_method returns an SSL_METHOD that avoids creating X509 objects for certificates. Additionally, SSL_CTX_set0_buffer_pool can be used to install a pool on an SSL_CTX so that certificates can be deduplicated across connections and across SSL_CTXs."
However my server uses OpenSSL is using a certificate created with x509 and is self-signed. I am not familiar with details of SSL certificate. From this post looks like x509 is required for self-signed.
Does it mean if I have to use self-signed certificate BoringSSL might not be an option here?
1) As PORTING.md goes on to say a few lines later
In order to use buffers, the application code also needs to implement its own certificate verification using SSL_[CTX_]set_custom_verify. Otherwise all connections will fail with a verification error. Auto-chaining is also disabled when using buffers.
so it looks like you need to do that. Note it says all, not just self-signed. Bummer.
2) Creating a self-signed cert with OpenSSL doesn't require the x509 subcommand. As the answers at your link correctly say, you can do two steps like
openssl req -newkey parms -keyout keyfile -out reqfile # create key and CSR
# if you want RSA with size per the config file, you can use -new (without value) instead
openssl x509 -req reqfile -signkey keyfile -out certfile # create cert from CSR
OR a single step like
openssl req -newkey parms -x509 -keyout keyfile -out certfile # create key and directly create cert
# ditto
# that's the DASH-x509 OPTION on the req COMMAND, not the x509 COMMAND
plus you can also use a DIFFERENT second step like
openssl req -newkey parms -keyout keyfile -out reqfile # create key and CSR
# ditto
openssl ca -in reqfile -selfsign -keyfile keyfile # create selfsigned cert from CSR
# but this also requires some other files to be set up and/or options added
and the result is the same. If you want to add extensions, and nowadays you often do especially for SSL/TLS, all three approaches can do so, but the details are slightly different, so to get it right you must look at the man page section(s) that are actually for what you are doing.
And there's even more options; you can create the key separately (and there's multiple options for that), and then use either req -new -key f (but not -newkey) followed by either x509 -req or ca -selfsign -in, or just req -new -x509 -key f (ditto).
But all of these end up with effectively the same cert, and the cert is not your problem, the BoringSSL API is.