pythongoogle-chromeselenium-webdriverproxyundetected-chromedriver

Python Selenium Undetected Chromedriver doesn't work with auth proxy


I use Undetected Chromedriver to bypass Cloudflare detection.

I trying force it to work with proxy, but unsuccessful. First attempt (IP just doesn't change):

import undetected_chromedriver as uc
options = uc.ChromeOptions()
...
ip,port,username,password = proxy.split(':')   #proxy in format ip:port:username:password
options.add_argument(f'--proxy-server=http://{username}:{password}@{ip}:{port}')
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)
driver.get('https://2ip.ru/')

Second attempt (with Selenium-Wire):

from seleniumwire import undetected_chromedriver as uc
...
ip,port,username,password = proxy.split(':')   #proxy in format ip:port:username:password
wire_options = {
        'proxy': {
            'http': f'http://{username}:{password}@{ip}:{port}', 
            'https': f'http://{username}:{password}@{ip}:{port}',
            'no_proxy': 'localhost,127.0.0.1' # excludes
        }  
    }
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options,seleniumwire_options=wire_options)
driver.get('https://2ip.ru/')

Browser trying to connect with proxy, but I need to manually enter password and username. I will be thanks, if you show me my mistakes or offer another decision of problem.


Solution

  • After day of searches on the Internet, I found out that Chrome doesn't support basic proxy authentification. Thank [AbdeLhalimSB][1] for leading me to the [right solution][2]. But this and other similar solutions don't work for me because my browser refuses to load extensions from zip archive. So, I needed to change this code so that browser can load extension from folder.

    extension.py:

    import os
    def proxies(username, password, endpoint, port,path):
        try:
            os.mkdir(path)
        except FileExistsError:
            pass
        manifest_json = """
        {
            "version": "1.0.0",
            "manifest_version": 2,
            "name": "Proxies",
            "permissions": [
                "proxy",
                "tabs",
                "unlimitedStorage",
                "storage",
                "<all_urls>",
                "webRequest",
                "webRequestBlocking"
            ],
            "background": {
                "scripts": ["background.js"]
            },
            "minimum_chrome_version":"22.0.0"
        }
        """
    
        background_js = """
        var config = {
                mode: "fixed_servers",
                rules: {
                singleProxy: {
                    scheme: "http",
                    host: "%s",
                    port: parseInt(%s)
                },
                bypassList: ["localhost"]
                }
            };
    
        chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});
    
        function callbackFn(details) {
            return {
                authCredentials: {
                    username: "%s",
                    password: "%s"
                }
            };
        }
    
        chrome.webRequest.onAuthRequired.addListener(
                    callbackFn,
                    {urls: ["<all_urls>"]},
                    ['blocking']
        );
        """ % (endpoint, port, username, password)
    
        with open(path+"\\manifest.json", 'w') as m_file:
            m_file.write(manifest_json)
        with open(path+"\\background.js", 'w') as b_file:
            b_file.write(background_js)
    

    main.py:

    from webdriver_manager.chrome import ChromeDriverManager
    import undetected_chromedriver as uc
    import os
    from extension import proxies
    options = uc.ChromeOptions()
    script_address = os.getcwd()
    ...
    ip,port,username,password = proxy.split(':')   #proxy in format ip:port:username:password
    proxies(username, password, ip, int(port),script_address+'\\Proxies\\'+ip)
    options.add_argument('--load-extension='+script_address+'\\Proxies\\'+ip)
    driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)
    

    Note, this solution will work in headless mode, but not in incognito mode. [1]: https://stackoverflow.com/users/20112601/abdelhalimsb [2]: How can I integrate proxies into my Selenium web scraper for True People Search?