phpfacebookweb-scrapingxpathhtml-parsing

Remotely Scrape Page and Get most Relevant title or Description for Images with XPath


What I'm looking at doing is essentially the same thing a Tweet button or Facebook Share / Like button does, and that is to scrape a page and the most relevant title for a piece of data. The best example I can think of is when you're on the front page of a website with many articles and you click a Facebook Like Button. It will then get the proper information for the post relative to (nearest) the Like button. Some sites have Open Graph tags, but some do not and it still works.

Since this is done remotely, I only have control of the data that I want to target. In this case the data are images. Rather than retrieving just the <title> of the page, I am looking to somehow traverse the dom in reverse from the starting point of each image, and find the nearest "title". The problem is that not all titles occur before an image. However, the chance of the image occurring after the title in this case seems fairly high. With that said, it is my hope to make it work well for nearly any site.

Thoughts:

Title backups:

Summary: Extracting the images isn't the problem, it's how to get relevant titles for them.

Question: How would you go about getting relevant titles for each of the images? Perhaps using DomDocument or XPath?


Solution

  • Your approach seems good enough, I would just give certain tags / attributes a weight and loop through them with XPath queries until I find something that exits and it's not void. Something like:

    i = 0
    
    while (//img[i][@src])
      if (//img[i][@alt])
        return alt
      else if (//img[i][@description])
        return description
      else if (//img[i]/../p[0])
        return p
      else
        return (//title)
    
      i++
    

    A simple XPath example (function ported from my framework):

    function ph_DOM($html, $xpath = null)
    {
        if (is_object($html) === true)
        {
            if (isset($xpath) === true)
            {
                $html = $html->xpath($xpath);
            }
    
            return $html;
        }
    
        else if (is_string($html) === true)
        {
            $dom = new DOMDocument();
    
            if (libxml_use_internal_errors(true) === true)
            {
                libxml_clear_errors();
            }
    
            if ($dom->loadHTML(ph()->Text->Unicode->mb_html_entities($html)) === true)
            {
                return ph_DOM(simplexml_import_dom($dom), $xpath);
            }
        }
    
        return false;
    }
    

    And the actual usage:

    $html = file_get_contents('http://en.wikipedia.org/wiki/Photography');
    
    print_r(ph_DOM($html, '//img')); // gets all images
    print_r(ph_DOM($html, '//img[@src]')); // gets all images that have a src
    print_r(ph_DOM($html, '//img[@src]/..')); // gets all images that have a src and their parent element
    print_r(ph_DOM($html, '//img[@src]/../..')); // and so on...
    print_r(ph_DOM($html, '//title')); // get the title of the page