python-3.xsslexceptioncertificateaiohttp

Python SSLCertVerificationError with aiohttp, but OpenSSL is fine

I'm trying to figure out why Python is throwing CERTIFICATE_VERIFY_FAILED exceptions for a certain endpoint I have configured, but other tools like OpenSSL / sslscan / sslyze seem to be fine with it.

The context of this is that we started receiving these errors around the same time we were rotating certificates. However, I have checked the certificate ordering (Actual cert + intermediates + root, in one file, in that order from top to bottom) and that is not the problem.

The simplest example I conjured up to test this is the following:

import certifi
import os
import socket
import ssl

SERVER = "myhost.example.com"
PORT = 443

context_instance = ssl.SSLContext()
context_instance.verify_mode = ssl.CERT_REQUIRED
context_instance.load_verify_locations(
    cafile=os.path.relpath(certifi.where()), capath=None, cadata=None
)
s = socket.socket()
ssl_socket = context_instance.wrap_socket(s)
ssl_socket.connect((SERVER, PORT))
print("Version of the SSL Protocol:", ssl_socket.version())
print("Cipher used:", ssl_socket.cipher())

For example, using facebook.com as the SERVER yields the following:

Version of the SSL Protocol: TLSv1.3
Cipher used: ('TLS_CHACHA20_POLY1305_SHA256', 'TLSv1.3', 256)

But when I use this to test our internal endpoint, I get the following error (I also get this when I use google.com also, which is weird):

Traceback (most recent call last):
  File "test.py", line 33, in <module>
    ssl_socket.connect((SERVER, PORT))
  File "/usr/local/lib/python3.8/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/usr/local/lib/python3.8/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)

My questions are:

  1. What's a foolproof way in Python 3 to verify if a TLS-enabled endpoint (let's just use HTTPS for now) is configured correctly?
  2. Is there a way to achieve a more verbose error output from Python to see what the problem on my internal endpoint is? The exception does not give any more details as to what could be wrong.

For reference, this is what I'm using:

And this is the output from OpenSSL:

$ openssl s_client -connect myhost.example.com:443 -4 <<< "Q"

CONNECTED(00000003)
depth=2 C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
verify return:1
depth=1 C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
verify return:1
depth=0 CN = *.example.com
verify return:1
---
Certificate chain
 0 s:CN = *.example.com
   i:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
 1 s:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
   i:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
 2 s:C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
   i:C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
 3 s:C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
   i:C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
---
Server certificate

<masked>

subject=CN = *.example.com

issuer=C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5687 bytes and written 409 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
Solution

  • For the future, there are a couple of insightful answers here, and all of them contribute to the overall answer:

    dave_thompson_085's answer was useful in determining that, if you want to check whether your TLS-enabled server endpoint is serving the right certificate, it's useful to run one of the following commands:

    For cases where the server only serves one hostname:

    $ openssl s_client -connect myhost.example.com:443 -4 <<< "Q"

    For cases where the server has multiple TLS hosts that it serves, and you want to check what happens when you don't pass a server hostname:

    $ openssl s_client -noservername -connect myhost.example.com:443 -4 <<< "Q"

    This will show a default backend if it's an HTTP server.

    For the Python bit, you need to pass in the server_hostname. Something like this was able to work for all valid HTTPS sites I tested:

    import os
import socket
import ssl

import certifi

SERVER = "myhost.example.com"
PORT = 443

context_instance = ssl.SSLContext()
context_instance.verify_mode = ssl.CERT_REQUIRED
context_instance.check_hostname = True
context_instance.load_verify_locations(
    cafile=os.path.relpath(certifi.where()), capath=None, cadata=None
)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_socket = context_instance.wrap_socket(s, server_hostname=SERVER)

ssl_socket.connect((SERVER, PORT))
print("Server hostname:", ssl_socket.server_hostname)
print("Version of the SSL Protocol:", ssl_socket.version())
print("Cipher used:", ssl_socket.cipher())

    Finally, the real underlying problem was the version of aiohttp==3.7.0 that we were using in our application. In the CHANGELOG for version 3.7.1, the following line shows the problem we were having:

    Fix a variable-shadowing bug causing ThreadedResolver.resolve to return the resolved IP as the hostname in each record, which prevented validation of HTTPS connections. #5110

    The script I ran to test the broken versions of aiohttp was the following:

    import asyncio

import aiohttp

SERVER = "myhost.example.com"


async def main():
    async with aiohttp.ClientSession(raise_for_status=True) as session:
        async with session.get(f"https://{SERVER}") as r:
            body = await r.json()
            print(body)


if __name__ == "__main__":
    asyncio.run(main())