Building Python and OpenSSL from source, but ssl module fails

I'm trying to build Python and OpenSSL from source in a container. Both seem to build correctly, but Python does not successfully create the _ssl module.

I've found a few guides online that say to un-comment and lines from Python-3.X.X/Modules/Setup and add the --openssldir=/usr/local/ssl flag to the ./configure step for OpenSSL. I do these in my dockerfile. This has had the effect that, during the ./configure output for Python, I see the following line.

checking for X509_VERIFY_PARAM_set1_host in libssl... yes

Yet I receive the following errors:

[91m*** WARNING: renaming "_ssl" since importing it failed: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by build/lib.linux-x86_64-3.8/_ssl.cpython-38-x86_64-linux-gnu.so)
[0m[91m*** WARNING: renaming "_hashlib" since importing it failed: /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1: version `OPENSSL_1_1_1' not found (required by build/lib.linux-x86_64-3.8/_hashlib.cpython-38-x86_64-linux-gnu.so)
[0m
Python build finished successfully!

...

Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl                                     


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

If ./configure finds X509..., why am I still getting the hashlib and ssl errors?

The full Dockerfile, FWIW:

FROM jenkins/jenkins:lts
USER root
RUN apt-get update && apt-get install -y apt-utils gcc make zlib1g-dev \
    build-essential libffi-dev checkinstall libsqlite3-dev 
RUN wget https://www.openssl.org/source/openssl-1.1.1d.tar.gz && \
    tar xzf openssl-1.1.1d.tar.gz && \
    cd openssl-1.1.1d && \
    ./config -Wl,--enable-new-dtags,-rpath,'$(LIBRPATH)' --prefix=/usr/local/ssl --openssldir=/usr/local/ssl && \
    make && \
    make test && \
    make install
RUN wget -q https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz && \
    tar -xzf Python-3.8.2.tgz && \
    cd Python-3.8.2 && \
    ./configure && \
    make && \
    make install
USER jenkins

Solution

The following worked for me on Amazon's EC2, with the default CentOS 7 system.

First, the openssl libraries on CentOS 7 are too old (Python 3.9+ wants openssl 1.1.1+ and the version available is 1.0.x). Install the newer ones:

sudo yum install openssl11-devel

Note: since writing this answer, Amazon has end-of-life'd the openssl11-devel package and updated openssl-devel to 3.0.8. 3.0.8 is more than enough for Python, so now you can just do yum install openssl-devel.

Unfortunately, CentOS doesn't actually put the SSL libraries anywhere that Python can find them. With some trial and error, I found that this makes ./configure happy:

export OPENSSL_LIBS=/usr/lib64/libssl.so
./configure \
  --with-openssl=/usr \
  --with-openssl-rpath=/usr/lib64 \
  --enable-optimizations

Explanation:

Python doesn't look in lib64 by default, which is where openssl11 is installed.
--with-openssl takes a path that ./configure appends include/openssl/ssl.h to, so make sure that is there for your system!

When you run ./configure, you're looking for a line near the end of the output like:

checking for stdlib extension module _ssl... yes

If you see missing instead of yes, search config.log for openssl, which should give you some guidance about where it's screwing up.

Hopefully this saves someone else the many hours I spent figuring this out.