I use Undetected Chromedriver to bypass Cloudflare detection.
I trying force it to work with proxy, but unsuccessful. First attempt (IP just doesn't change):
import undetected_chromedriver as uc
options = uc.ChromeOptions()
...
ip,port,username,password = proxy.split(':') #proxy in format ip:port:username:password
options.add_argument(f'--proxy-server=http://{username}:{password}@{ip}:{port}')
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)
driver.get('https://2ip.ru/')
Second attempt (with Selenium-Wire):
from seleniumwire import undetected_chromedriver as uc
...
ip,port,username,password = proxy.split(':') #proxy in format ip:port:username:password
wire_options = {
'proxy': {
'http': f'http://{username}:{password}@{ip}:{port}',
'https': f'http://{username}:{password}@{ip}:{port}',
'no_proxy': 'localhost,127.0.0.1' # excludes
}
}
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options,seleniumwire_options=wire_options)
driver.get('https://2ip.ru/')
Browser trying to connect with proxy, but I need to manually enter password and username. I will be thanks, if you show me my mistakes or offer another decision of problem.
After day of searches on the Internet, I found out that Chrome doesn't support basic proxy authentification. Thank [AbdeLhalimSB][1] for leading me to the [right solution][2]. But this and other similar solutions don't work for me because my browser refuses to load extensions from zip archive. So, I needed to change this code so that browser can load extension from folder.
extension.py:
import os
def proxies(username, password, endpoint, port,path):
try:
os.mkdir(path)
except FileExistsError:
pass
manifest_json = """
{
"version": "1.0.0",
"manifest_version": 2,
"name": "Proxies",
"permissions": [
"proxy",
"tabs",
"unlimitedStorage",
"storage",
"<all_urls>",
"webRequest",
"webRequestBlocking"
],
"background": {
"scripts": ["background.js"]
},
"minimum_chrome_version":"22.0.0"
}
"""
background_js = """
var config = {
mode: "fixed_servers",
rules: {
singleProxy: {
scheme: "http",
host: "%s",
port: parseInt(%s)
},
bypassList: ["localhost"]
}
};
chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});
function callbackFn(details) {
return {
authCredentials: {
username: "%s",
password: "%s"
}
};
}
chrome.webRequest.onAuthRequired.addListener(
callbackFn,
{urls: ["<all_urls>"]},
['blocking']
);
""" % (endpoint, port, username, password)
with open(path+"\\manifest.json", 'w') as m_file:
m_file.write(manifest_json)
with open(path+"\\background.js", 'w') as b_file:
b_file.write(background_js)
main.py:
from webdriver_manager.chrome import ChromeDriverManager
import undetected_chromedriver as uc
import os
from extension import proxies
options = uc.ChromeOptions()
script_address = os.getcwd()
...
ip,port,username,password = proxy.split(':') #proxy in format ip:port:username:password
proxies(username, password, ip, int(port),script_address+'\\Proxies\\'+ip)
options.add_argument('--load-extension='+script_address+'\\Proxies\\'+ip)
driver = uc.Chrome(executable_path=ChromeDriverManager().install(), options=options)
Note, this solution will work in headless mode, but not in incognito mode. [1]: https://stackoverflow.com/users/20112601/abdelhalimsb [2]: How can I integrate proxies into my Selenium web scraper for True People Search?