web-scrapingmashape

Accessing Metacritic API and/or Scraping


Does anybody know where documentation for the Metacritic API is/if it still works? The Metacritic API at https://market.mashape.com/byroredux/metacritic-v2#get-user-details disappeared today.

I'm scraping the site myself but keeping getting blocked by a 429 Slow down. I got data 3 times this hour and haven't been able to get anymore in 20 minutes which is making testing difficult and application possibly useless. Let me know if there's anything else I can be doing.


Solution

  • I had to add a user agent like JCDJulian said and now it allows me to scrape. So for Ruby:

    agent = Mechanize.new
    agent.user_agent_alias = "Mac Firefox"
    

    Then it stopped giving me the 403 Forbidden error.