scrapycommand-line-argumentszyte

How can I add a new spider arg to my own template in Scrapy/Zyte


I am working on a paid proxy spider template and would like the ability to pass in a new argument on the command line for a Scrapy crawler. How can I do that?


Solution

  • This is achievable by using kwargs in your spider's __init__-Method:

    import scrapy
    
    
    class YourSpider(scrapy.Spider):
        name = your_spider
    
        def __init__(self, *args, **kwargs):
            super(YourSpider, self).__init__(*args, **kwargs)
            self.your_arg = kwargs.get("your_cmd_arg", 42)
    

    Now it would be possible to call the spider as follows:
    scrapy crawl your_spider -a your_cmd_arg=foo

    For more information on the topic, feel free to check this page in the Scrapy documentation.