This piece of code creates a memory leak. I'm not sure why. This code is shortedned from a longer method that polls a redis list for messages (which returns either a message or null immidiately if there's no message) and then waits for sometime before polling again. but I have taken out the distracting pieces.
async pollQ() {
while (true) {
await new Promise((resolve) =>
setTimeout(resolve, 100 + 100 * Math.random()),
);
}
}
In consecutive heap snapshots, with this method body, there's an object like {created, parent}
with a large positive delta (in thousands) that grows with time and chains similar objects like this:
Each of these objects look like this:
If I comment out the wait-while loop, the heap stays stable.
This is the memory consumption graph from ecs - where the graph starts growing upwards is when this code was deployed. (the x axis scale is in days, 7 days total) Orange is max usage, green is median, blue is min.
I've also tried using a setTimeout version that repeatedly calls pollQ at the end of the method. That produces the same result when there are other await calls (the redis calls).
async pollQ() {
// here we await other async functions (redis library calls)
setTimeout(this.pollQ.bind(this), 100 + 100 * Math.random());
}
I've tried using a driver function in the hope that when the pollQ
scope is done, things will be gc-ed. But this had the same issue.
const driver = (() => {
this.pollQ().finally(() => {
setTimeout(driver, 100 + 100 * Math.random())
})
}).bind(this)
driver()
Only doing a setInterval(pollQ, <inteval>)
works without the heap growing incessantly but that's not ideal given the process might not complete in the interval. I'm out of ideas at this point and more importantly I don't understand the leak. The code doesn't look like it would create a leak but the heap keeps growing. Any help is appreciated.
It is part of a larger Nestjs application. The complete pollQ method is this:
async pollQ() {
let message: RedisMessage;
while (true) {
try {
message = await this.redisQueue.receiveMessage();
if (message) {
await this.process(message);
await this.redisQueue.deleteMessage(message.id);
}
} catch (e) {
this.logger.error(`Error polling redis q: ${e}`);
if ((message._attempts || 0) < 3) {
await this.redisQueue.returnToQueue(message.id);
} else {
await this.redisQueue.deleteMessage(message.id);
}
} finally {
await new Promise((resolve) =>
setTimeout(resolve, 100 + 100 * Math.random()),
);
}
}
}
redisQueue.receiveMessage()
returns immidiately regardless of whether the queue has a message. pollQ
is called from onModuleInit
which is a Nestjs specific lifecycle hook method for di container managed objects. The container creates an application scoped instance of the enclosing class.
async onModuleInit() {
this.pollQ();
}
The process
method picks some fields from the event and persist it in a table.
The application is running on nodejs v20.19.0.
I finally have the full picture thanks to a colleague's hawk eyes.
In the codebase, AsyncLocalStorage is really from the npm package async-local-storage
here.
This package uses node's async hooks to create a new context (a vanilla js object) on every async operation start and keeps a reference to the parent's storage via the parent async resource's execution context id. You can see this here. Because we do this on a loop infinitely, this keeps piling up, impossible for the GC to clean up.
One way to get rid of the chaining would be to call scope() on every iteration.
I finally have half an answer why this was happening. In the codebase, AsyncLocalStorage
is enabled. This creates a context for each async call and chains past contexts for consecutive calls. I can reproduce this behavior with this isolated snippet:
import { enable } from 'async-local-storage';
enable();
async function main() {
while (true) {
await new Promise((resolve) =>
setTimeout(resolve, 100 + 100 * Math.random()),
);
}
}
main()
What I want to understand is why the contexts are being chained since this promises here get resolved before creating the next one.
I have settled for a setInterval based approach that waits for the processing code to finish before running it again.
const driver = (() => {
if (!this.polling) this.pollQ();
}).bind(this);
setInterval(driver, 100 + 100 * Math.random());
and
async pollQ() {
this.polling = true;
try {
// awaits
} catch (e) {
// awaits
} finally {
this.polling = false;
}
}
This heap is stable with this. But I'd still like to know why the original code grows the heap non stop.