phpweb-scrapingphantomjscasperjs

CasperJS/PhantomJS much slower than Curl

When I tried curl www.yelp.com it takes 1.1 secs. However retrieving the page using CasperJS takes over a minute!

Is this normal? How do I find out what's slowing casper/phantom down? I am suspecting its some HTTP redirects that casper is not following?

var casper = require('casper').create();
var url = 'http://www.yelp.com';

casper.start(url);
casper.then(function() {
    console.log(  this.getHTML() );
    this.exit();
});

casper.run();

enter image description here

Solution

  • Are you on Windows? If yes, there is a mysterious network problem when automatic proxy is being used. See the release notes for more details: http://phantomjs.org/release-1.9.html.

    In general, try to analyze the network requests and responses. A very simple way to trace the network traffic:

    page.onResourceRequested = function (request) {
  console.log('Request ' + JSON.stringify(request, undefined, 4));
};
page.onResourceReceived = function (response) {
  console.log('Receive ' + JSON.stringify(response, undefined, 4));
};

    You need to tweak it further if you want the timing etc. Read the documentation on this Network Monitoring features.