pythonselenium-webdriverhttp-proxygoogle-chrome-headless

How to set proxies with Selenium options when connecting to command-line generated google-chrome

I am successfully running the chrome browser from the terminal with the following command:

google-chrome --remote-debugging-port=9222 --user-data-dir="C:\selenum\ChromeProfile"

I am then able to connect to this browser instance with the following line of code when initialising the driver:

options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")

I would now like to set a proxy server address in the format of "http://username:password@proxyhost:port".

I am usually able to use proxies just fine with Selenium using a manifest_json block and adding to options i.e.:

pluginfile = 'proxy_auth_plugin.zip'
with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)
    zp.writestr("background.js", background_js)
options.add_extension(pluginfile)

However, when the above experimental debugger option (above) is added to the code, the proxies no longer work and the system IP address is used.

I have also tried setting a proxy from the command-line with '--proxy-server="http://username:password@proxyhost:port', but as far as I understand, Chrome doesn't accent username and passwords through command-line.

So I'm wondering is there any way to achieve setting proxies with Selenium after connecting to a command-line generated Chrome instance?

Thanks in advance for your help.

Solution

  • When you launch Chrome from the command-line, you'll need to specify an extension that sets authenticated proxy settings. To create that extension, use the info provided in https://stackoverflow.com/a/35293284/7058266.

    To specify the extension when launching Chrome, use --load-extension=path/to/extension (https://stackoverflow.com/a/22198816/7058266). In your case, that might look like this:

    google-chrome --remote-debugging-port=9222 --load-extension=path/to/extension

    There's a Python framework called SeleniumBase that already does those things. Here's the Python code that creates a proxy-with-auth extension. The SeleniumBase UC Mode code that launches Chrome before attaching chromedriver to it can be found here. (Used for evading bot-detection services.)