pythonselenium-webdriverwebautomation

Running multiple instances of selenium is generating connection error


I am running selenium automation scripts using python3, on linux and undetected_chrome.
I am also using proxies.
When I run only one instances it works fine and does the job however the moment a second instance is launched I get this error :

2024-04-30 00:09:28 - ERROR - HTTPConnectionPool(host='localhost',
port=44073): Max retries exceeded with url:
/session/2488e939aa54121cb42b165948495489/actions (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7debd7c86fb0>: Failed to establish a new connection: [Errno 111]
Connection refused'))

Despite the proxies are changing with each driver.

I have trying all kinds of solutions I could think of around it.


Solution

  • This solution uses SeleniumBase UC Mode instead of undetected-chromedriver to spin up multiple simultaneous web browsers with stealth: (pip install seleniumbase)

    import sys
    from concurrent.futures import ThreadPoolExecutor
    from seleniumbase import Driver
    sys.argv.append("-n")  # Activate SB thread-locking as needed
    
    def launch_driver(url):
        driver = Driver(uc=True)
        try:
            driver.uc_open_with_reconnect(url, 4)
            driver.sleep(2)
        finally:
            driver.quit()
    
    urls = ['https://gitlab.com/users/sign_in' for i in range(3)]
    with ThreadPoolExecutor(max_workers=len(urls)) as executor:
        for url in urls:
            executor.submit(launch_driver, url)
    

    Be sure to swap the example URL (that uses bot-detection software) with the URL that you want.