ERR_SSL_VERSION_OR_CIPHER_MISMATCH when using self-signed certificates with network IPs

I am attempting to create self-signed certificates in order to test some SSL functionality locally. To create the self-signed certificates I've performed the steps below:

Step 1. Create Root crt / key for self-signing

openssl req -nodes -x509 -sha256 -days 1825 -newkey rsa:2048 -keyout rootCA.key -out rootCA.crt

Step 2. Create CSR using openssl

openssl req -nodes -newkey rsa:2048 -keyout domain.key -out domain.csr

Step 3. Create a domain file with subject alt names for the cert

manually created file domain.ext with contents below:

authorityKeyIdentifier=keyid,issuer 
basicConstraints=CA:FALSE 
subjectAltName = @alt_names 
[alt_names] 
DNS.1 = localhost 
DNS.2 = desktop-3dqn06e 
DNS.3 = 192.168.56.1.nip.io 
IP.1 = 192.168.56.1

Step 4. Generate self-signed certificate using Root crt/key files

openssl x509 -req -CA rootCA.crt -CAkey rootCA.key -in domain.csr -out domain.crt -days 1825 -CAcreateserial -extfile domain.ext

Step 5. Run basic express web-server to test the certificates

var path = require('path');
var fs = require('fs');
var express = require('express');
var http = require('http');
let https = require('https');
var tls = require('tls');

// create new express app and assign it to `app` constant
const app = express();

/***
 * Start the server: listens on http://localhost:${PORT}
 */
 //SSL credentials
 function getCredentials(){
     let privateKey  = fs.readFileSync('/path/to/domain.key', 'utf8');
     let certificate = fs.readFileSync('/path/to/domain.crt'), 'utf8');
     let certauth = fs.readFileSync('/path/to/rootCA.crt'), 'utf8');
     let credentials = {key: privateKey, cert: certificate, ca: certauth};
     return credentials;
 }
 var ctx = function() { return tls.createSecureContext(getCredentials()) };

 //Start HTTPS server
 const httpsServer = https.createServer({
     SNICallback: (servername, cb) => cb(null, ctx())
 }, app);
 httpsServer.listen(443);

 //Redirect HTTP requests to HTTPS
 const httpServer = http.createServer(function (req, res) {
    res.writeHead(301, { "Location": "https://" + req.headers['host'] + req.url });
    res.end();
 });
 httpServer.listen(80);

 //Send home page
 app.get('/',function(req,res) {
    res.sendFile(path.join(__dirname+'/index.html'));
 });

When I attempt client requests to the server via a browser using either localhost or the machine name, the connection is fine and I can see the connection is secure

However, if I attempt to access using the network IP configured in domain.ext, the connection fails with ERR_SSL_VERSION_OR_CIPHER_MISMATCH Furthermore, the nip.io equivalent address also fails with DNS_PROBE_FINISHED_NXDOMAIN

Solution

... if I attempt to access using the network IP configured in domain.ext, the connection fails with ERR_SSL_VERSION_OR_CIPHER_MISMATCH

You only provide a SSL context in SNICallback. But with just using an IP address SNI will not be used and thus the SNICallback not called. Thus no context with certificates is known which means only ciphers which don't need certificates are supported by the server which means no cipher overlap which what the client provides (i.e. only ciphers needing a certificate).

Thus you need to provide the arguments constructed in getCredentials also for createServer, so that it has a useful context when called without SNI.

Furthermore, the nip.io equivalent address also fails with DNS_PROBE_FINISHED_NXDOMAIN

This is a DNS problem, i.e. happens before the TCP connection can be created and this means also before TLS handshake is done. This is therefore irrelevant to the TLS problem. I have no idea why it fails, the DNS lookup works for me.