node.jscheerionode-request

Async web requests are making 'socket hangup' in node.js

I am writing a code that 'for' loops a large number of time to scrape web pages. Code is like:

var request = require('request');
for(i=0; i<10000; i++){
//request goes here, with processing in its callback
}

But this causes socket hangup. Then I tried sync request to do this then code works fine and processes request by request but this is making execution hell slow. Please guide how can large number of web requests can be processed for scraping for example.

Solution

  • You should look at option pool when send a request using request module.

    There are 3 options:

    Examples:

    for (var i = 0; i < 10000; i++) {
   // make a request
   request({
      pool: false,
      // other options
   }, function(err, res, body) {
       // handle response here
   });
}

    References: