I'm using Redis to store data from an API with a TTL of a 7 days.
Here's some psuedo of what I'm doing to set the data.
First I check if the data is in Redis.
for(let i=0; i< requestedData.length; i++){
let exists = await client.exists(cacheKey);
if(!exists) {
// queue to be fetched
}
}
After the data is fetched I store it.
for(let i=0; i< data.length; i++){
await client.set(cacheKey, JSON.stringify(dataToStore), {
EX: 604800,
});
}
This seems to work well as when I validate the cache keys through Redis-cli everything is there. The problem comes the next time I attempt to get the data from Redis.
Redis will inconsistently have data disappear from the cache. It's not overwritten, it's just gone. Normally I'd expect ~800 keys, but there will randomly be 450, 300, 575, etc. Even though the only time the cache is used at all in subsequent requests, it's to check if the data already exists (which it does) as shown above in the psuedo. I've debugged for hours trying to figure out what's going on and I'm wondering if it's something out of my control.
Does exists()
remove keys sometimes?
I'm using an M1 macbook.. is there an issue with compatibility with Redis?
I've checked if max_memory is being hit and its only about 10% used.
If anyone has had this issue before and has managed to figure it out your input would be greatly appreciated!
I found out what was happening. Turns out because I had maxmemory=120mb
the keys were being evicted after I had already called exists()
and wrote new keys to the cache evicting the the least recently used keys.
Figured this out but observing the evicted_keys
stat after every data request. Simply increasing maxmemory
solved the issue.