pythonselenium-webdriverproxyselenium-grid

How to run selenium standalone behind a proxy?


Several days ago I bumped into a need of using Selenium webdriver.Remote with proxy auth. I found few articles that enlighten this topic just a bit. So I would like to share my solution. I hope someone might well find it usefull.

Python 3.10.11

Package requirements: selenium==4.9.0 # On futher versions of Selenium it won't work due to API changes selenium-wire==5.1.0


Solution

  • from selenium.webdriver import ChromeOptions
    from seleniumwire import webdriver
    
    
    PROXY_USERNAME = "USERNAME"
    PROXY_PASSWORD = "PASSWORD"
    CURRENT_MACHINE_IP = "YOUR_CURRENT_MACHINE_IP"
    BACKEND_PORT = 8087  # any awailable port
    
    options = ChromeOptions()
    options.add_argument(f"--proxy-server={CURRENT_MACHINE_IP}:{BACKEND_PORT}")
    options.add_argument("--ignore-certificate-errors")
    
    seleniumwire_options = {
        "addr": CURRENT_MACHINE_IP,
        "auto_config": False,
        "proxy": {
            "http": f"http://{PROXY_USERNAME}:{PROXY_PASSWORD}@PROXY_LINK:PROXY_PORT",
            "https": f"http://{PROXY_USERNAME}:{PROXY_PASSWORD}@PROXY_LINK:PROXY_PORT",
            "no_proxy": "localhost,127.0.0.1",
            "verify_ssl": False,
        },
        "port": BACKEND_PORT,
    }
    
    driver = webdriver.Remote(
        command_executor="http://127.0.0.1:4444",
        desired_capabilities=options.to_capabilities(),
        seleniumwire_options=seleniumwire_options,
    )
    
    try:
        driver.get("https://python.org/")
        with open("test.html", "w", encoding="utf-8") as file:
            file.write(driver.page_source)
    finally:
        driver.quit()
    

    docker-compose.yaml

    version: '3.8'
      services:
        selenium:
          image: selenium/standalone-chrome:latest
          environment:
            - SE_NODE_SESSION_TIMEOUT=30
            - SE_NODE_MAX_SESSIONS=5
            - SE_NODE_OVERRIDE_MAX_SESSIONS=true
          ports:
            - 4444:4444