GoLang Mochi MQTT Server with Python Paho MQTT Client TLS Authentication Verification Failed

I am trying to use the GoLang Mochi MQTT server on Windows 11 (https://github.com/mochi-mqtt/server) with the Python Paho (https://pypi.org/project/paho-mqtt/) client for testing on the same macine. The authentication type I'd like to use is TLS (https://github.com/mochi-mqtt/server/blob/main/examples/tls/main.go).

I setup a local PKI infrastructure on the Windows 11 machine with the Python PKI library (https://github.com/BrixIT/Python-PKI).

I create a root CA with the PKI library and 2 client certificates.

developer@docker-desktop:Python-PKI$ python3 pypki.py list
+---------------+----------------+---------+---------------------+----------+
| Common name   | Organisation   | State   | Expiration date     |   Serial |
|---------------+----------------+---------+---------------------+----------|
| Dev Cert 1    | Me Co.         | Valid   | 2034-05-01 18:52:04 |     1000 |
| Dev Cert 2    | Me Co.         | Valid   | 2034-05-01 18:54:12 |     1001 |
+---------------+----------------+---------+---------------------+----------+

I copied the client certificates to both the Python Paho test client directory as well as the GoLang MQTT server directory.

I slightly modified the Mochi MQTT server example for TLS to be the following.

package main

import (
    "crypto/tls"
    "crypto/x509"
    "github.com/mochi-mqtt/server/v2/packets"
    "io/ioutil"
    "log"
    "os"
    "os/signal"
    "syscall"

    mqtt "github.com/mochi-mqtt/server/v2"
    "github.com/mochi-mqtt/server/v2/hooks/auth"
    "github.com/mochi-mqtt/server/v2/listeners"
)

func main() {
    sigs := make(chan os.Signal, 1)
    done := make(chan bool, 1)
    signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)
    go func() {
        <-sigs
        done <- true
    }()

    cer, loadCertErr := tls.LoadX509KeyPair("Dev Cert 2.cert.pem", "Dev Cert 2.key.pem")
    if loadCertErr != nil {
        log.Println(loadCertErr)
        os.Exit(1)
    }

    // Optionally, if you want clients to authenticate only with certs issued by your CA,
    // you might want to use something like this:
    certPool := x509.NewCertPool()
    caCertPem, certErr := ioutil.ReadFile("ca.cert.pem")
    if certErr != nil {
        log.Println("Error: Failed to subscribe to direct topic", "error", certErr)
        os.Exit(1)
    }
    _ = certPool.AppendCertsFromPEM(caCertPem)
    tlsConfig := &tls.Config{
        ClientCAs:    certPool,
        ClientAuth:   tls.RequireAndVerifyClientCert,
        Certificates: []tls.Certificate{cer},
    }

    server := mqtt.New(&mqtt.Options{
        InlineClient: true,
    })
    _ = server.AddHook(new(auth.AllowHook), nil)

    log.Println("Listening on :1883")
    tcp := listeners.NewTCP(listeners.Config{
        ID:        "t1",
        Address:   ":1883",
        TLSConfig: tlsConfig,
    })
    err := server.AddListener(tcp)
    if err != nil {
        log.Fatal(err)
    }

    log.Println("Listening on :1882")
    ws := listeners.NewWebsocket(listeners.Config{
        ID:        "ws1",
        Address:   ":1882",
        TLSConfig: tlsConfig,
    })
    err = server.AddListener(ws)
    if err != nil {
        log.Fatal(err)
    }

    log.Println("Listening on :8080")
    stats := listeners.NewHTTPStats(
        listeners.Config{
            ID:        "stats",
            Address:   ":8080",
            TLSConfig: tlsConfig,
        }, server.Info,
    )

    err = server.AddListener(stats)
    if err != nil {
        log.Fatal(err)
    }

    log.Println("Running server")
    go func() {
        err := server.Serve()
        if err != nil {
            log.Fatal(err)
        }
    }()

    // Inline subscribe
    callbackFn := func(cl *mqtt.Client, sub packets.Subscription, pk packets.Packet) {
        server.Log.Info("inline client received message from subscription", "client", cl.ID, "subscriptionId", sub.Identifier, "topic", pk.TopicName, "payload", string(pk.Payload))
    }
    subscribeErr := server.Subscribe("direct/#", 1, callbackFn)
    if subscribeErr != nil {
        server.Log.Error("Error: Failed to subscribe to direct topic", "error", subscribeErr)
        os.Exit(1)
    }

    // Inline publish
    publishErr := server.Publish("direct/publish", []byte("packet scheduled message"), false, 0)
    if publishErr != nil {
        server.Log.Error("Error: Failed to publish to direct topic", "error", publishErr)
        os.Exit(1)
    }

    <-done
    server.Log.Warn("caught signal, stopping...")
    _ = server.Close()
    server.Log.Info("main.go finished")
}

I also modified the Paho MQTT client example to connect to the MQTT server. This is all done on the same Windows 11 machine.

import time
import paho.mqtt.client as paho
import ssl
import pathlib
#define callbacks
def on_message(client, userdata, message):
  print("received message =",str(message.payload.decode("utf-8")))

def on_log(client, userdata, level, buf):
  print("log: ",buf)

def on_connect(client, userdata, flags, rc):
  print("publishing ")
  client.publish("muthu","muthupavithran",)


client=paho.Client()
client.on_message=on_message
client.on_log=on_log
client.on_connect=on_connect
print("connecting to broker")
client.tls_set(str(pathlib.Path().cwd() / pathlib.Path("Python-PKI/certs/Dev Cert 1.cert.pem")), tls_version=ssl.PROTOCOL_TLSv1_2)
client.tls_insecure_set(True)
client.connect("localhost", 1883, 60)

##start loop to process received messages
client.loop_start()
#wait to allow publish and logging and exit
time.sleep(1)

When the GoLang server runs and the Paho test client connects, the server prints the following errors:

time=2024-05-04T13:11:03.354-05:00 level=INFO msg="added hook" hook=allow-all-auth
time=2024-05-04T13:11:03.357-05:00 level=INFO msg="attached listener" id=t1 protocol=tcp address=[::]:1883
time=2024-05-04T13:11:03.357-05:00 level=INFO msg="attached listener" id=ws1 protocol=wss address=:1882
time=2024-05-04T13:11:03.357-05:00 level=INFO msg="attached listener" id=stats protocol=https address=:8080
time=2024-05-04T13:11:03.357-05:00 level=INFO msg="inline client received message from subscription" client=inline subscriptionId=1 topic=direct/publish payload="packet scheduled message"
time=2024-05-04T13:11:03.357-05:00 level=INFO msg="mochi mqtt starting" version=2.6.3
time=2024-05-04T13:11:03.359-05:00 level=INFO msg="mochi mqtt server started"
2024/05/04 13:11:03 Listening on :1883
2024/05/04 13:11:03 Listening on :1882
2024/05/04 13:11:03 Listening on :8080
2024/05/04 13:11:03 Running server
time=2024-05-04T13:11:20.865-05:00 level=WARN msg="" listener=t1 error="read connection: remote error: tls: unknown certificate authority"

Also, the Paho client prints the following:

C:\Users\me\go-mqtt-server\test\client.py:17: DeprecationWarning: Callback API version 1 is deprecated, update to latest version
  client=paho.Client()
connecting to broker
Traceback (most recent call last):
  File "C:\Users\me\go-mqtt-server\test\client.py", line 24, in <module>
    client.connect("localhost", 1883, 60)
  File "C:\Users\me\go-mqtt-server\test\Lib\site-packages\paho\mqtt\client.py", line 1435, in connect
    return self.reconnect()
           ^^^^^^^^^^^^^^^^
  File "C:\Users\me\go-mqtt-server\test\Lib\site-packages\paho\mqtt\client.py", line 1598, in reconnect
    self._sock = self._create_socket()
                 ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\go-mqtt-server\test\Lib\site-packages\paho\mqtt\client.py", line 4612, in _create_socket
    sock = self._ssl_wrap_socket(sock)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\go-mqtt-server\test\Lib\site-packages\paho\mqtt\client.py", line 4671, in _ssl_wrap_socket
    ssl_sock.do_handshake()
  File "C:\Users\me\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)

I'd like to have the client only be able to connect with a certificate signed by the local PKI CA. What am I doing wrong and how can I fix it? Thank you.

Solution

Based on the statement "authentication type I'd like to use is TLS "I believe your intention is to use self-signed certificates for two purposes:

Server verification (allows the client to confirm that the server is who you expect it to be) - the server is using Dev Cert 2.cert for this.
Client authentication - I believe this is your intention, but the client is not passing a certificate so am unsure (will assume you intend to use Dev Cert 1 to authenticate the client).

Both of the certificates are signed by the same CA which can make this a bit confusing (in production it's probably better to use separate CA's). So in this case the server (to verify the clients certificate), and the client (to verify the servers certificate), need to trust the CA (which you created with Python-PKI).

Now you say "I copied the client certificates to both the Python Paho test client directory". I'm going to assume this means you also copied ca.cert.pem, which is referenced in your Go code (if not, you will need to do this), and that this is the Python-PKI generated CA certificate (meaning that, at first glance, the server looks OK).

As an aside - it's probably beneficial to do this step by step. i.e.

Setup mochi-server with a server certificate, test with something like MQTTX and then python.
Add the requirement for a client certificate in mochi-server, test as above.

Currently it's quite possible there are issues with both the client and server, so its good to test things piece by piece.

Anyway in your python code you call :

client.tls_set(str(pathlib.Path().cwd() / pathlib.Path("Python-PKI/certs/Dev Cert 1.cert.pem")), tls_version=ssl.PROTOCOL_TLSv1_2)
client.tls_insecure_set(True)

There are a few issues here:

tls_set is only being passed ca_certs (the first argument). This means no client certificate will be sent to the server (so that part would fail if it were reached).
The certificate being passed as ca_certs did not sign the servers certificate (both certificates were signed by the same CA)

So, I think, you want something like (abbreviated):

client.tls_set("Python-PKI/certs/ca.cert.pem", "Python-PKI/certs/Dev Cert 1.cert.pem", "Python-PKI/certs/Dev Cert 1.cert.key", tls_version=ssl.PROTOCOL_TLSv1_2)
client.tls_insecure_set(True)

Note that tls_insecure_set(True) should not really be needed if you are issuing these certificates yourself (you can set the CN to localhost). This setting only controls (simplifying a bit) whether the client confirms the CN in the server certificate matches the hostname (the certificate will still be checked to ensure its valid and issued by a trusted CA). You can change this with the cert_reqs paramater if needed (default is ssl.CERT_REQUIRED).

Hopefully the above helps, if not then please try to simplify your question (e.g. focus on the server and test with MQTTX) as there are a lot of variables here.

Update: a few comments on why a separate CA would be used for both server and client? The goal here is for me to create a closed system where only clients using a Certificate that my own CA signs can authenticate with the MQTT Server

The client and server certificates fill different roles and, as I see it, the only benefit to using the same CA is that it simplifies things (slightly!). Here are a few reasons why separating them might be a good idea:

Security
- Least Privilege: It's likely the client CA will be used fairly frequently (as you add clients) whereas the server CA will be rarely used (how often do you add a new server?). This means you may need to share access to one (to support staff etc) but not the other (much harder if they are the same CA).
- Isolation: If one CA is compromised the other remains unaffected (in reality if either are compromised in your scenario you have issues!).
- When using a single CA, if someone can generate client certs then they can also, potentially, impersonate your server (because a client cert can be used on the server). The reverse also applies.
You may want to use a trusted third party CA for the server (e.g. lets encrypt) to avoid the need to distribute your own CA cert.
Clarity - using two CA's makes it easier to understand the trust model (i.e. in your python you would have a server-ca.pem which makes it clear that this is used to validate the server certificate).

That's a few thoughts off the top of my head.