python-2.7web-scrapinggoogle-shopping-api

Should I switch from Mechanise to Scrapy or BeautifulSoup?

I'm trying to build a web scraper to scrape Google shopping results. Any ideas where to start? I was using mechanize with python but I've found it has some limitations. Would Scrapy or BeautifulSoup be better?

Solution

  • Posted on behalf of the OP.

    I found this article to be helpful :

    http://swizec.com/blog/scraping-with-mechanize-and-beautifulsoup/swizec/5039

    I was getting throttling from the Google side.