I've got a 'script' that does thousands of requests to a specific API. This API will only allow 5 requests per second (and probably it measures differently then me). To make the requests I'm using request-promise
framework, and I've superseded the normal request-promise
function with this:
const request_promise = require('request-promise')
function waitRetryPromise() {
var count = 0 // keeps count of requests
function rp(options) {
const timedCall = (resolve) => setTimeout( ()=>resolve(rp(options)),1000) // recursive call
count += 1
if (count % 3 == 0) { // recalls after a second on every third request
return new Promise(timedCall)
} else {
return request_promise(options)
}
}
return rp
}
const rp = waitRetryPromise()
Once around 300 requests (give or take) are fired off in short succession, these requests start to interfere with each other. Does anyone have a better solution? I thought the recursive call to this same function would help, and It did but it didn't solve the problem. Maybe there is a pattern to queue requests, and do them a few at a time? A library perhaps?
Thanks!
OK, rather than recursing the call to rp etc, just make sure you delay between requests by an appropriate amount ... for 5 per second, that's 200ms
function waitRetryPromise() {
let promise = Promise.resolve();
return function rp(options) {
return promise = promise
.then(() => new Promise(resolve => setTimeout(resolve, 200)))
.then(() => request_promise(options));
}
}
const rp = waitRetryPromise();