[SOLVED] Connection to google.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely

Connection to google.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely

While running my test scripts with selenium == 4.2.0 like this:

from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType


options = FirefoxOptions()
service = Service()
options.headless = True
options.accept_insecure_certs = True 
proxy = Proxy({
    'httpProxy': proxy_addr,
    'sslProxy': proxy_addr,
    'proxyType': ProxyType.MANUAL
})

options.proxy = proxy

wd = Firefox(service=service, options=options)
wd.execute("get", {'url': 'http://google.com'})

I'm getting the following error:

An error occurred during a connection to www.google.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site. Please contact the website owners to inform them of this problem. This website might not support the TLS 1.2 protocol, which is the minimum version supported by Firefox. Enabling TLS 1.0 and TLS 1.1 might allow this connection to succeed.

I think the problem is that I'm using a proxy which is running on localhost. Since I use the browser in headless mode and configured accept_insecure_certs = True I don't see how to find a workaround for this error. I'll be grateful if someone advises what else I can try to do.

Solution

I'm assuming that you're using a MITM that allows you to intercept the TLS traffic. If so, then this is exactly the scenario that HSTS preload is intended to prevent ;)

Your MITM will be generating a fake certificate on the fly, but because it does not match the HSTS preload list that is baked into the browser, then this is why you get presented with an error (rather than a dialog that asks if you want to continue).

You may be able to get around this by configuring the proxy to strip the HSTS header on all responses (check the documentation for the particular MITM that you are using).