I'm using React and fetch in the client to make requests to the Discogs API. In this API, there's a limit of max 60 request per minute. For managing this Discogs is adding custom values like "remaining requests", "used requests" or "maximum allowed requests", on the response headers but due to cors those headers cannot be readed.
So what I decided to do is to create a request wrapper for this API, from where I could:
I've managed to do a working example using a singleton Object where the jobs are queued and managed with setTimeout
function to delay the call of the request.
This works for me when using simple callbacks, but I don't know how to return a value to the React component and how to implement it with Promises instead of callbacks (fetch).
I also don't know how to cancel the timeout or the fetch request from the react component.
You can check this example, where I've simplified it. I know that maybe that's not the best way to do it or maybe this code is shit. That's why any help or guidance on it would be very much appreciated.
I wanted to limit the number of requests but also put them on hold until it is allowed by the API, so I though that the best option was to run them sequentially in a FIFO order, with a delay of 1 sec between them so I do not exceed the 60 requests in 1 minute requirement. I was also thinking about let them run some of them concurrently, but in this case the waiting time could be high once the limit is reached.
I created then 2 things:
A 'useDiscogsFetch' hook
useDiscogsFetch.js
import { useEffect, useRef, useState } from 'react';
import DiscogsQueue from '@/utils/DiscogsQueue';
import { v4 as uuidv4 } from 'uuid';
const useDiscogsFetch = (url, fetcher) => {
const [data, setData] = useState(null);
const [error, setError] = useState(null);
const requestId = useRef();
const cancel = () => {
DiscogsQueue.removeRequest(requestId.current);
}
useEffect(() => {
requestId.current = uuidv4();
const fetchData = async () => {
try {
const data = await DiscogsQueue.pushRequest(
async () => await fetcher(url),
requestId.current
);
setData(data)
} catch (e) {
setError(e);
}
};
fetchData();
return () => {
cancel();
};
}, [url, fetcher]);
return {
data,
loading: !data && !error,
error,
cancel,
};
};
export default useDiscogsFetch;
A DiscogsQueue singleton class
DiscogsQueue.js
class DiscogsQueue {
constructor() {
this.queue = [];
this.MAX_CALLS = 60;
this.TIME_WINDOW = 1 * 60 * 1000; // min * seg * ms
this.processing = false;
}
pushRequest = (promise, requestId) => {
return new Promise((resolve, reject) => {
// Add the promise to the queue.
this.queue.push({
requestId,
promise,
resolve,
reject,
});
// If the queue is not being processed, we process it.
if (!this.processing) {
this.processing = true;
setTimeout(() => {
this.processQueue();
}, this.TIME_WINDOW / this.MAX_CALLS);
}
}
);
};
processQueue = () => {
const item = this.queue.shift();
try {
// Pull first item in the queue and run the request.
const data = item.promise();
item.resolve(data);
if (this.queue.length > 0) {
this.processing = true;
setTimeout(() => {
this.processQueue();
}, this.TIME_WINDOW / this.MAX_CALLS);
} else {
this.processing = false;
}
} catch (e) {
item.reject(e);
}
};
removeRequest = (requestId) => {
// We delete the promise from the queue using the given id.
this.queue.some((item, index) => {
if (item.requestId === requestId) {
this.queue.splice(index, 1);
return true;
}
});
}
}
const instance = new DiscogsQueue();
Object.freeze(DiscogsQueue);
export default instance;
I don't know if it's the best solution but it gets the job done.