javascriptnode.jspromiserequest-promise

Node.js - Working with an API limit of 5 requests per second


I've got a 'script' that does thousands of requests to a specific API. This API will only allow 5 requests per second (and probably it measures differently then me). To make the requests I'm using request-promise framework, and I've superseded the normal request-promise function with this:

const request_promise  = require('request-promise')

function waitRetryPromise() {
  var count = 0 // keeps count of requests
  function rp(options) {
    const timedCall = (resolve) => setTimeout( ()=>resolve(rp(options)),1000) // recursive call
    count += 1
    if (count % 3 == 0) { // recalls after a second on every third request
      return new Promise(timedCall)
    } else {
      return request_promise(options)
    }
  }
  return rp
}

const rp = waitRetryPromise()

Once around 300 requests (give or take) are fired off in short succession, these requests start to interfere with each other. Does anyone have a better solution? I thought the recursive call to this same function would help, and It did but it didn't solve the problem. Maybe there is a pattern to queue requests, and do them a few at a time? A library perhaps?

Thanks!


Solution

  • OK, rather than recursing the call to rp etc, just make sure you delay between requests by an appropriate amount ... for 5 per second, that's 200ms

    function waitRetryPromise() {
        let promise = Promise.resolve();
        return function rp(options) {
            return promise = promise
            .then(() => new Promise(resolve => setTimeout(resolve, 200)))
            .then(() => request_promise(options));
        }
    }
    const rp = waitRetryPromise();