I am writing a program which needs to iterate through a number of items, performing an HTTP GET request on each item in turn. The HTTP GET request is rate-limited by the third-party API to a maximum of 5 requests per minute. I am trying to use Bottleneck to limit my requests out to this API, to little success.
A minimum example of my problem is as follows:
// index.ts
import * as API from "./lib/API";
(async () => {
try {
//Gather all requests
const list_of_requests = await API.Requests.get_list_of_all_requests(...);
// Fire off each request at the appropriate rate
list_of_requests.forEach(async (request) => {
try {
const request_result = await API.ResultGetter.get_result(...);
} catch (error) {
console.log(error.message);
}
});
} catch (error) {
console.log(error.name + ": " + error.message);
}
})();
Within my ./lib/API
:
// API.ts
import { AxiosRequestConfig } from "axios";
import { scheduleRequest } from "../util/Limiter";
export async function ResultGetter(...) {
const requestURL: string = "https://www.website.ext/...";
const requestConfig: AxiosRequestConfig = {
url: requestURL,
method: "GET",
responseType: "json",
};
return await scheduleRequest(requestConfig);
}
And finally, in my util/Limiter
// Limiter.ts
import Bottleneck from "bottleneck";
import axios, { AxiosRequestConfig, AxiosResponse } from "axios";
/*
5 API requests per minute; 500 API requests per day
*/
export const limiter = new Bottleneck({
reservoir: 500,
reservoirRefreshAmount: 500,
reservoirRefreshInterval: 24 * 60 * 60 * 1000,
maxConcurrent: 1,
mintime: Math.ceil(1 / (5 / 60 / 1000)),
});
export async function scheduleRequest(
request: AxiosRequestConfig
): Promise<AxiosResponse<any>> {
return limiter.schedule(() => axios(request));
}
However, when I actually utilize this program the requests fire off at the fastest rate possible. I'm truly not sure what I'm missing in this case. To my understanding, limiter.schedule(...)
should be returning a promise that resolves when the request happens, limited by the rate limit set. But that is not what's happening here.
Turns out it was just a difficult to find typo.
In my original code, I had the following to define my rate limiter:
export const limiter = new Bottleneck({
reservoir: 500,
reservoirRefreshAmount: 500,
reservoirRefreshInterval: 24 * 60 * 60 * 1000,
maxConcurrent: 1,
mintime: Math.ceil(1 / (5 / 60 / 1000)),
});
Well the fact that I wrote mintime
instead of minTime
with a capital T
made all the difference in the world.