pythongoogle-chromepyppeteer

net::SOCKS_CONNECTION_FAILED when routing Pyppeteer request through socks proxy


Here's my attempt at a minimal webscraper using pyppeteer 1.0.2 routing through a NordVPN socks proxy:

import asyncio
from pyppeteer import launch

import json
import requests

res = requests.get('https://nordvpn.com/api/server')
servers = json.loads(res.content)
socks_servers = [server['ip_address'] for server in servers if server['features']['socks']]
socks_server = socks_servers[0]
port = '1080'

async def with_proxy():
    browser = await launch(
        executablePath='/usr/bin/google-chrome',
        headless=True,
        args=[f'--proxy-server=socks5://{socks_server}:{port}']
    )

    page = await browser.newPage()
    await page.authenticate({
        'username': username,
        'password': password
    })
    await page.goto('https://api.myip.com')
    await browser.close()

await with_proxy()

However, this results in the following error:

---------------------------------------------------------------------------
PageError                                 Traceback (most recent call last)
Cell In[33], line 23
     20     await page.goto('https://api.myip.com')
     21     await browser.close()
---> 23 await with_proxy()

Cell In[33], line 20, in with_proxy()
     15 # do not forget to put "await" before async functions
     16 await page.authenticate({
     17     'username': username,
     18     'password': password
     19 })
---> 20 await page.goto('https://api.myip.com')
     21 await browser.close()

File /usr/local/lib/python3.11/site-packages/pyppeteer/page.py:831, in Page.goto(self, url, options, **kwargs)
    829 result = await self._navigate(url, referrer)
    830 if result is not None:
--> 831     raise PageError(result)
    832 result = await watcher.navigationPromise()
    833 watcher.cancel()

PageError: net::ERR_SOCKS_CONNECTION_FAILED at https://api.myip.com

As a test, I tried the same thing through the native requests library and it seemed to work fine:

import requests

def native_requests():
    with requests.Session() as s:
        s.headers['Connection'] = 'close'
        prox = {'https':f'socks5://{username}:{password}@{socks_server}:{port}'}
        r = s.get('https://api.myip.com', proxies=prox)
        print(r)
        s.close()

native_requests()

This prints <Response [200]> as expected. What am I doing wrong here?


Solution

  • There is no issue with your code - it should work fine if proxy and port is valid.

    I was unable to connect to NordVPN socks proxy via provided server and port, however I was able to connect to random socks5 proxy from Free-Proxy

    Video with proof (on the first try connection failed which is ok for free proxy, but on second try it was succeed)

    So as a suggestion - try to use proxy list that you trust or implement retry mechanism (if connection failed - try to launch with next proxy from list, etc.)

    server = '198.8.94.174'
    port1 = '39078'
    async def with_proxy():
        browser = await launch(
            {
                "headless": False,
                "ignoreHTTPSErrors": True,
                'args': ['--start-maximized',
                         f'--proxy-server=socks5://{server}:{port1}',
                         "--ignore-certificate-errors",
                         "--ignore-certificate-errors-spki-list"],
                'executablePath': '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
            }
        )
    
        page = await browser.newPage()
        await page.goto('https://api.myip.com')
        await browser.close()
    
    asyncio.get_event_loop().run_until_complete(with_proxy())