proxyscrapydeferredscrapydproxytunnel

Scrapyd Retry if Errback in twisted deferreds / Could not open CONNECT tunnel Error

I'm working with a cycle of proxies in Scrapy, sometimes some proxies have errors and that's stopping my spider.. The error is "Could not open CONNECT tunnel" How do I change the code to retry with another proxy in case of this error?

Here's the code that needs to be modified: https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/handlers/http11.py

Solution

  • Something like this:

    from scrapy.core.downloader.handlers.http11 import TunnelError


class RetryMiddleware(RetryMiddleware):
    def process_exception(self, request, exception, spider):
        if ( isinstance(exception, self.EXCEPTIONS_TO_RETRY) or isinstance(exception, TunnelError) ) \
                and 'dont_retry' not in request.meta:
            return self._retry(request, exception, spider)

    In settings.py:

    DOWNLOADER_MIDDLEWARES = { 
   'myproject.myretry.RetryMiddleware': 200,
   'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': None
}