linuxscrapy

How to create file name of Feed Export dynamically using scrapy command

This is command to run my scraper.

scrapy crawl monitor -a filename="ScrapeProject.csv" -o filename.csv

It runs and saves the scraped data into filename.csv

I want to schedule as cronjob I want filename.csv to be the current datetime the scraper ran.

I tried with back-ticks but didnt work

scrapy crawl monitor -a filename="ScrapeProject.csv" -o `date`.csv

Also tried like that

scrapy crawl monitor -a filename="ScrapeProject.csv" -o "date".csv

EDIT:

Below is the command I ran upon @dps recommendation but it prompts me to enter something? 

root@ubuntu:/home/mani/pricemonitor# scrapy crawl monitor -a filename="ScrapeProject.csv" -o `date +\%m`.`date +\%d`.`date +\%y`.csv`
> 
>
Solution

  • Scrapy Feed Exports also understand (some built-in) storage URI parameters out of the box.

    %(time)s is one of them.

    So you can do something like:

    scrapy crawl monitor -a filename="ScrapeProject.csv" -o '%(time)s.csv'

    which will create output files in the form YYYY-mm-ddTHH-MM-SS, e.g. 2017-05-11T12-12-18.csv.

    Internally, time is converted using datetime.utcnow().replace(microsecond=0).isoformat().replace(':', '-').

    Note: you can use any spider attribute in your Feed URI (what you set with -o). Remember that any spider argument (stuff you can add to the command line with -a key=value) will be available as spider argument (as strings).