javascriptnode.jsx-ray

Node x-ray crawling data from collection of url


I'm trying to scrape a list in a site that leads to other pages that has the same formatting.

I was able to create a collection of all the a tags, but when I try to visit a collection of pages, the key I try to create with it doesn't get added in my returned object.

Here's an example of what I'm trying to do with stack overflow:

var Xray = require('x-ray');
var x = Xray();
x('http://stackoverflow.com/', {
    title: x(['a@href'], 'title'),
}) (function(err, obj) {
    console.log(obj);
});

I'm expecting my obj.title to be a list of titles of all the a href pages, instead I just get an empty object.

However if I were to try just using the first a href then I get the title no problem.

var Xray = require('x-ray');
var x = Xray();
x('http://stackoverflow.com/', {
    title: x('a@href', 'title'),
}) (function(err, obj) {
    console.log(obj);
});

Has anyone run into this problem before?


Solution

  • I ran into that problem before and my solution goes like this:

    var Xray = require('x-ray');
    var x = Xray();
    x('http://stackoverflow.com/', {
        title: x('a', [{links:'@href'}])
    }) (function(err, obj) {
        obj.forEach(function(links.link) {
            x(links.link, "title")(function(err, data){
                    console.log(data) // should print the title
            });
    });
    

    Let me know if you run into any problems.