[SOLVED] Dealing with rate limiting APIs in NodeJS

Dealing with rate limiting APIs in NodeJS

So I need a way to run a function x amount of times in a given second, then wait until the next second to run the next set. I'm using the Yelp Fusion API to call a 'search' https://api.yelp.com/v3/search and then running a details query on each of the search results: https://api.yelp.com/v3/businesses/business_id.

So the first search query returns an array of businesses like this:

const businesses = response.body.businesses
for (business of businesses) {
    fetchPlaceDetails(business.id)
}

Theoretically I can run fetchPlaceDetails() in an async manner, but going through 50+ results takes too long. Instead, I'd like to run the query maybe 5 times, wait a second (to get past the rate limit), run the next 5, wait a second, run the next 5, etc.

Not sure how to implement this but I figure it's got to be a pretty standard issue to have when using APIs with rate limits. Also, I'm not sure but nowhere in the documentation do I see an actual call/second limit specified, but I'd assume it's maybe 10 requests/second or so?

Solution

I'd like to run the query maybe 5 times, wait a second (to get past the rate limit), run the next 5, wait a second, run the next 5, etc.

Then just do that:

const {businesses} = response.body;
for (let i=0; i<businesses.length; i+=5) {
    await Promise.all([
        ...businesses.slice(i, i+5).map(({id}) => fetchPlaceDetails(id)),
        new Promise(resolve => { setTimeout(resolve, 1000); })
    ]);
}

This will also ensure that you wait more than one second between the batches if the 5 requests take longer than that.