I am scraping the following site using requests and certifi: https://ecf.ared.uscourts.gov/
When I scrape it on my local machine, it works fine, but when I run automated tests in Travis, it fails with:
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
The code I'm using for this is pretty straightforward:
url = 'https://ecf.ared.uscourts.gov/cgi-bin/login.pl'
r = requests.post(
url,
verify=certifi.where(),
timeout=60,
files={
'login': ('', username),
'key': ('', password)
},
)
Any ideas what's different between Travis and my local machine that would cause a certificate failure? I thought the idea of certifi was to standardize all this. (I've verified I have updated versions on both computers.)
Got it! Travis uses an old version of OpenSSL which doesn't support cross-signed certificates, but my local dev machine uses a modern version of OpenSSL.
The solution here is to use certifi.old_where()
which provides older, less secure certificate support. I've resigned myself to this until I can upgrade the version of OpenSSL that we have.