Here's my attempt at a minimal webscraper using pyppeteer 1.0.2 routing through a NordVPN socks proxy:
import asyncio
from pyppeteer import launch
import json
import requests
res = requests.get('https://nordvpn.com/api/server')
servers = json.loads(res.content)
socks_servers = [server['ip_address'] for server in servers if server['features']['socks']]
socks_server = socks_servers[0]
port = '1080'
async def with_proxy():
browser = await launch(
executablePath='/usr/bin/google-chrome',
headless=True,
args=[f'--proxy-server=socks5://{socks_server}:{port}']
)
page = await browser.newPage()
await page.authenticate({
'username': username,
'password': password
})
await page.goto('https://api.myip.com')
await browser.close()
await with_proxy()
However, this results in the following error:
---------------------------------------------------------------------------
PageError Traceback (most recent call last)
Cell In[33], line 23
20 await page.goto('https://api.myip.com')
21 await browser.close()
---> 23 await with_proxy()
Cell In[33], line 20, in with_proxy()
15 # do not forget to put "await" before async functions
16 await page.authenticate({
17 'username': username,
18 'password': password
19 })
---> 20 await page.goto('https://api.myip.com')
21 await browser.close()
File /usr/local/lib/python3.11/site-packages/pyppeteer/page.py:831, in Page.goto(self, url, options, **kwargs)
829 result = await self._navigate(url, referrer)
830 if result is not None:
--> 831 raise PageError(result)
832 result = await watcher.navigationPromise()
833 watcher.cancel()
PageError: net::ERR_SOCKS_CONNECTION_FAILED at https://api.myip.com
As a test, I tried the same thing through the native requests
library and it seemed to work fine:
import requests
def native_requests():
with requests.Session() as s:
s.headers['Connection'] = 'close'
prox = {'https':f'socks5://{username}:{password}@{socks_server}:{port}'}
r = s.get('https://api.myip.com', proxies=prox)
print(r)
s.close()
native_requests()
This prints <Response [200]>
as expected. What am I doing wrong here?
There is no issue with your code - it should work fine if proxy and port is valid.
I was unable to connect to NordVPN socks proxy via provided server and port, however I was able to connect to random socks5 proxy from Free-Proxy
Video with proof (on the first try connection failed which is ok for free proxy, but on second try it was succeed)
So as a suggestion - try to use proxy list that you trust or implement retry mechanism (if connection failed - try to launch with next proxy from list, etc.)
server = '198.8.94.174'
port1 = '39078'
async def with_proxy():
browser = await launch(
{
"headless": False,
"ignoreHTTPSErrors": True,
'args': ['--start-maximized',
f'--proxy-server=socks5://{server}:{port1}',
"--ignore-certificate-errors",
"--ignore-certificate-errors-spki-list"],
'executablePath': '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
}
)
page = await browser.newPage()
await page.goto('https://api.myip.com')
await browser.close()
asyncio.get_event_loop().run_until_complete(with_proxy())