ruby-on-railsrubynokogiri

extracting XML from HTML?

Reference page

The XML is embedded under the <pre> tag of the returned HTML page. I can extract the contents of the <pre> tag, but I am unable to convert this to XML correctly. I tried using the to_xml method of the NodeSet class, but it seems that the line endings (\n) are messing up the parsing.

Here is a snippet of my code:

url = "http://www.ncbi.nlm.nih.gov/pubmed/?term=NS044283[GR]&dispmax=200&report=xml"
doc = Nokogiri::XML(open(url))
pre = doc.xpath('//pre')
xml = pre.to_xml
contents = Nokogiri::XML(xml)
articles = contents.xpath('\\PubmedArticle')
(article = [])
Solution

  • Since you're going to use Nokogiri to parse it anyway, just call content instead of to_xml:

    require 'nokogiri'
require 'open-uri'
url = "http://www.ncbi.nlm.nih.gov/pubmed/?term=NS044283[GR]&dispmax=200&report=xml"
doc = Nokogiri::XML(open(url))
pre = doc.xpath('//pre')
xml = "<root>" + pre.text + "</root>"
contents = Nokogiri::XML(xml)
articles = contents.css('PubmedArticle')
puts contents.css('ArticleTitle').map{|x| x.content}.count   
=> 25