Unable to connect to MongoDB server with authentication

we are trying to install and configure MongoDB (version 7.0) on our Ubuntu server. We used this document as our guide. This is what we did step by step (after having installed MongoDB server on the Ubuntu machine) from command line:

1 – Connect to one of the hosts and generate a new private key using openssl:

openssl genrsa -out mongoCA.key -aes256 8192

2 – Sign a new CA certificate:

openssl req -x509 -new -extensions v3_ca -key mongoCA.key -days 365 -out mongoCA.crt

3 – Issue self-signed certificates for all nodes

openssl req -new -nodes -newkey rsa:4096 -keyout mongossl1.key -out mongossl1.csr

openssl x509 -CA mongoCA.crt -CAkey mongoCA.key -CAcreateserial -req -days 365 -in mongossl1.csr -out mongossl1.crt

cat mongossl1.key mongossl1.crt > psmdb1.pem

This is our MongoDB configuration on our Ubuntu host machine:

net:   
  ssl:
    mode: requireSSL
    PEMKeyFile: /etc/mongodb/ssl/psmdb1.pem
    CAFile: /etc/mongodb/ssl/mongoCA.crt 
security:  
  authorization: enabled  
  clusterAuthMode: x509

Then we restart the MongoDB server on Ubuntu host.

Then we start mongo-shell (mongosh) on the Ubuntu host machine (the same one where MongoDB was just now configured and restarted), it is able to connect to that MongoDB instance fine.

However, when trying to connect from outside the mongo shell, either in the same Ubuntu host, or some other VM, through some client like PyMongo or even Compass, we are getting errors.

mongo --tls --tlsCAFile /etc/ssl/new/mongoCA.crt --tlsCertificateKeyFile /etc/ssl/new/psmdb1.pem --authenticationMechanism MONGODB-X509 --authenticationDatabase '$external' --host <our Ubuntu host public IP address> --port 27017

Any idea how to fix this?

You mixed up several topics.

security.clusterAuthMode: x509 is used for internal membership authentication in a Sharded Cluster and/or Replica Set. It looks like, you use a MongoDB Stand-alone instance, thus it does not apply.

In MongoDB you can use a server certificate to encrypt the connection. The server certificate is also used to guarantee that you connect to the correct server.

A client can authenticate with username/password or a client certificate.

In principle it is possible to use one single certificate for all, however it does not make much sense. Better create two certificates, one for the server and one for the client. You must also create the x.509 user. Have a look at Use x.509 Certificates to Authenticate Clients and How Security in MongoDB works (using x.509 cert) to get a generic overview.

Update

Based on your comment, your config-file is almost correct.

As already mentioned security.clusterAuthMode is not relevant on a Stand-alone MongoDB, you can skip it. You should use tls options instead of deprecated ssl options:

net:
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/mongodb/ssl/psmdb1.pem
    CAFile: /etc/mongodb/ssl/mongoCA.crt
security:  
  authorization: enabled  

When you connect to MongoDB then you have to use these parameters

mongosh --tls --tlsCAFile /etc/ssl/new/mongoCA.crt --tlsCertificateKeyFile /etc/ssl/new/psmdb1.pem --authenticationDatabase 'admin' --user 'username' --password 'secret' --host <our Ubuntu host public IP address> --port 27017

Legacy mongo shell mongo has been removed in MongoDB version 6.0. Port 27017 is the default, thus you can skip it.

Options --authenticationMechanism MONGODB-X509 --authenticationDatabase '$external' are used when you like to authenticate the client with x.509 certificate - which you are not asking for.

Read/write privileges are defined granted roles in createUser command. IP ranges you can restrict with parameter authenticationRestrictions, for example

use admin
db.runCommand(
   {
     createUser: "api",
     pwd: "secret"
     roles: [
       { role: "dbOwner", db: "application" }
     ],
     authenticationRestrictions: [
        { clientSource: [ "10.20.30.0/24" ]
     ]
   }
)

As I already mentioned, I would suggest to create dedicated client and server certificate, see extendedKeyUsage