node.jsnode-request

Requesting URL's one by one


I'm trying to get some data from a lot of URLs using 'request', but I can't manage to do it one url at a time.

I've tried to understand async/promises in order to make it work, but with no success. Trial and error didn't work.

I've seen other methods using different modules, but this requires rewriting most of my code and I believe there is an easier way to adapt it to my current code.

Here is a minimized version of the code :

const request = require('request');
const fs = require('fs');
const prod = fs.readFileSync('prod.txt', "utf8");
const prodid = prod.split("|");
var i;
var summary=[];


for (i=0;i<prodid.length;i++){
request('https://www.api.example.com/id='+prodid[i], { json: true }, (err, res, body) => {
  if (err) { return console.log(err); }
if (body == 'NULL') {
   console.log("Page " + i + " out of " + prodid.length + " is NULL!");
} else {
summary.push(body.items[0].Name);
summary.push(body.items[0].ISOnr);
summary.push(body.items[0].GTIN);
console.log("Page " + i + " out of " + prodid.length + " is done!");
fs.appendFileSync('data.txt',JSON.stringify(summary));
}

});
}

There is no async/promise involved in the example above, just the requests inside a loop.

From what I've seen, when I get the results, there is no particular order (probably is the order of which finishes first).

In the console, I always see page 500 out of 500, not 1/500, 2/500, etc.

What I'm trying to achieve, is by making each request in the order of URLs (preferably with a 1000ms delay between them)


Solution

  • You can promisify your request:

    for (i = 0; i < prodid.length; i++) {
        const result = await new Promise((resolve, reject) =>
            request(
                'https://www.api.example.com/id=' + prodid[i],
                { json: true },
                (err, res, body) => {
                    if (err) {
                        reject(err);
                    }
                    if (body == 'NULL') {
                        console.log('Page ' + i + ' out of ' + prodid.length + ' is NULL!');
                    } else {
                        resolve(body);
                    }
                }
            )
        );
        if (result) {
            summary.push(result.items[0].Name);
            summary.push(result.items[0].ISOnr);
            summary.push(result.items[0].GTIN);
            console.log('Page ' + i + ' out of ' + prodid.length + ' is done!');
            fs.appendFileSync('data.txt', JSON.stringify(summary));
        }
    }