javascriptweb-scrapingcasperjs

casper evaluate modifying global var


I'm new to CasperJS and I'm having issues with the evaluate() function. I have the following sample code:

var results = [];  // a global var
var links = [link1, link2 ,....];  //a list of urls
var current = 0;

function search(url){
  this.start(url, function(){
    results = this.evaluate(function() { 
      var returnVal = [];
      //some code to store data into returnVal
      return returnVal;
      };
  })
}

function check(){
  if(current<7){
    this.echo('---Page' + (current + 1) + '---');
    search.call(this, links[current]);
    console.log(results.length);
    this.run(check);
    current++;
  } else{
    this.exit();
}

casper.start()
casper.then(function(){this.echo("starting");});
casper.run(check);

SO basically the code is supposed to gather some data from 7 pages and store the data in the global var results. Each page has 50 data to be stored. So the expected output is:

Starting
---Page1---
50
---Page2---
100
---Page3---
150
...

But the actual output is:

Starting
---Page1---
0
---Page2---
50
---Page3---
50
...

I'm expecting the length of results to increase after each iteration. However when I ran it, the length of results stayed the same, which means after the first iteration, it never got changed. Also, I don't understand why the length is 0 on page1. I couldn't figure out why that is.


Solution

  • I think this is because, every time the test runs, the global 'results' is being reassigned. You would need to append to the array instead of reassigning it after each step. Something like

    var tempResults = this.evaluate(function() {
        var returnVal = [];
        // load your data from the page
        return returnVal;
    };
    
    
    for(var i=0, j=tempResults.length; i < j; i++) {
        //append to global here
        results.push(tempResults[i]);
    }