TLS/SSL error connecting to simple MQTT Aedes Server

I have a requirement to implementing a custom MQTT Broker using Aedes.

To start I'm building a simple beginner's code to enable MQTT client tools to connect using TLS/SSL (I've succedeed without SSL/TLS):

import Aedes from "aedes";
import fs from "fs";
import tls from "tls";

const port = process.env.MQTT_PORT || 8883; // MQTT Port
const id = "TEST_IOT";

const options = {}
options.key = fs.readFileSync('./src/security/cert.key');
options.cert = fs.readFileSync('./src/security/cert.pem');

const aedes = new Aedes({
  id: id,
});

aedes.on("client", client => {
  console.log(
    "Client Connected: \x1b[33m" + (client ? client.id : client) + "\x1b[0m",
    "to broker",
    aedes.id
  );
});

aedes.on("ping", (packet, client) => {
  console.log(
    `PING : MQTT Client ${client ? client.id : client
    } ping to aedes broker ${aedes.id}`
  );
})


const server = tls.createServer(aedes.handle, options);

server.listen(port, () => {
  console.log(`Find the server at: mqtt://localhost:${port}/`);
});

This is how I've generated the cert files:

I've created a ref.cfg file:

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = BR
ST = São Paulo
L = São Paulo
O = Acme Software
OU = Acme
CN = localhost
[v3_req]
keyUsage = critical, digitalSignature, keyAgreement
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = www.localhost.com
DNS.2 = localhost.com
DNS.3 = localhost

And then to generate the certificates:

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout cert.key -out cert.pem -config req.cnf -sha256

Server runs fine, no errors, no logs.

I'm using a client MQTTX tool running on the same machine - MacOs running Monterrey 12.5.1 - to test the server.

Trying to connect generates the following error on MQTTX:

  at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:82:16)
[2023-03-02 08:12:00] [INFO] Connect close, MQTT.js onClose trigger
[2023-03-02 08:12:13] [INFO] Connect client, MQTT/SSL connection: mqtts://localhost:8883
[2023-03-02 08:12:13] [INFO] MQTTX client with ID 9b922306-4789-4cbd-bb3b-591077a1fc46 assigned
[2023-03-02 08:12:13] [ERROR] Connect fail, MQTT.js onError trigger, Error: write EPROTO 140320376334904:error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE:../../third_party/boringssl/src/ssl/tls_record.cc:587:SSL alert number 40
140320376334904:error:1000009a:SSL routines:OPENSSL_internal:HANDSHAKE_FAILURE_ON_CLIENT_HELLO:../../third_party/boringssl/src/ssl/handshake.cc:579:

My MQTTX configuration:

Trying to debug, got some hints to use curl (understood that even having no https service, it would first try to handshake the TLS/SSL protocol) - indeed I got a handshake error algo using curl:

% curl -v https://localhost:8883
*   Trying 127.0.0.1:8883...
* Connected to localhost (127.0.0.1) port 8883 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
* Closing connection 0
curl: (35) error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure

I guess this is related to certificate installation, version or configuration, but I cannot find out what is wrong. I have tried reinstalling openssl, change req.cfg parameters several times with no success.

How can I fix this and make my simple SSL/TLS connection go true?

TLS/SSL has 2 tasks:

1. Provide a mechanism to allow the data transmitted between 2 parties to be encrypted.
2. Ensures that one or both ends of the connection are who the claim to be.

In most normal cases it is the server that needs to prove it is who it claims to be, it does that by presenting a certificate that has been signed by a trusted party (A Certificate Authority).

In your case you have not provided a trusted party to say the broker certificate should be trusted (because the certificate is signed by it's own private key).

So for your client to trust your Self Signed Certificate it also needs a copy of the brokers certificate.

When not working with Self Signed Certificates, clients will come with a list of public trusted CA certificates to check certificates against.

As mentioned in the comments, you need to either tell the client to not validate the broker certificate or supply the certificate to use to validate.

It is always possible to tell the clients not to carry out task 2, but that removes all the protection from somebody running a man in the middle attack. e.g. there appears to be a "Self Signed" option MQTTX image you have shared.

(Client certificates are only needed if the broker needs to prove the client is who it claims to be)