rubyproxyweb-scrapingscraperwiki

Proxy / Fetch data from other countries


Certain websites require us to have a particular IP address to display certain information eg. ads for country X. I would like to know if it is possible to use a proxy (preferably ruby one) with my ruby script @scraperwiki to get the results as if I was in that country X. Right now the script gets the results in the UK and if I use an HTTP proxy I can see the website that I want to retrieve the data from correctly. The problem is Scraperwiki does not return the webpage like if it was in country X


Solution

  • Yes. You should be using Mechanize:

    require 'mechanize'
    agent = Mechanize.new
    agent.set_proxy host, port
    page = agent.get url
    

    Now call page#search or page#at just like you would with your Nokogiri document.