asp.netweb-scrapingvisual-studio-2008

Yahoo Web Scrapes: What are the limits?


We are using a web scraper and have it set up to have a sleep function which has a random function set up (so that it isn't the same time between each scrape) but we are still getting blocked from Yahoo after 20-30 requests.

Does any one know if there is a limit (i.e: 20 requests per minutes, 200 an hour) Right now our average between each request is around 3-6 seconds. Thanks for any help


Solution

  • So the answer is 5000 queries. Taken from

    http://forums.digitalpoint.com/showthread.php?t=736784

    http:// developer. yahoo. com/search/rate.html