I am using Open::URI and Nokogiri to scrape a Google search page:
require 'open-uri'
require 'nokogiri'
url = http://www.google.co.uk/search?&q=toys&start=0&num=&complete=0
doc = Nokogiri::HTML(open(url))
mas = doc.css('li.g')[7]
mas.at_css('.mas-row')
From this result I am interested in just one result:
"Amazon.co.uk: Toys - Harry Potter: Toys & Games"
and I would like to get the data from "div class mas-row".
I can not find it. I looked in the "doc" variable and it can not be found. After that I looked for the text that is in that "div" and for the first div a part of the text was found but nothing from the next div.
Can anyone help me with this?
The div with mas-row is included in the HTML. It's rendered by JavaScript.
Use a library that can handle JavaScript, such as selenium.