Question:
How can proxy scrapy
requests with socks5
?
I know I can use
polipo
to convertSocks
Proxy ToHttp
Proxy
But:
I want to set a Middleware or some changes in scrapy.Request
import scrapy
class BaseSpider(scrapy.Spider):
"""a base class that implements major functionality for crawling application"""
start_urls = ('https://google.com')
def start_requests(self):
proxies = {
'http': 'socks5://127.0.0.1:1080',
'https': 'socks5://127.0.0.1:1080'
}
for url in self.start_urls:
yield scrapy.Request(
url=url,
callback=self.parse,
meta={'proxy': proxies} # proxy should be string not dict
)
def parse(self, response):
# do ...
pass
what should I assign to proxies
variable?
It is currently not possible. There is a feature request for it.