rubyanemone

Error in fetching a list of urls from a website using anemone

Code:

require 'anemone'
Anemone.crawl("http://www.example.com/") do |anemone|
  anemone.on_every_page do |page|
    puts page.url
  end
end

When I try this code I should get a list of all the urls on that website but all I get is just the name of the website. What can possibly be the error and how do I get a list of all the urls?

Solution

  • I guess anemone just can't follow redirects or something like this, cause "http://example.com" redirects me on other site. Have you tried to crawl other sites? http://stackoverflow.com, for example.