javascriptnode.jsasync-awaitmemory-leaksasync-hooks

JS Infinite loop with await creates memory leak


This piece of code creates a memory leak. I'm not sure why. This code is shortedned from a longer method that polls a redis list for messages (which returns either a message or null immidiately if there's no message) and then waits for sometime before polling again. but I have taken out the distracting pieces.

async pollQ() {
    while (true) {
        await new Promise((resolve) =>
          setTimeout(resolve, 100 + 100 * Math.random()),
        );
    }
  }

In consecutive heap snapshots, with this method body, there's an object like {created, parent} with a large positive delta (in thousands) that grows with time and chains similar objects like this:

chain ob retained objects

Each of these objects look like this:

object structure

If I comment out the wait-while loop, the heap stays stable.

This is the memory consumption graph from ecs - where the graph starts growing upwards is when this code was deployed. (the x axis scale is in days, 7 days total) Orange is max usage, green is median, blue is min.

ecs memory usage graph

I've also tried using a setTimeout version that repeatedly calls pollQ at the end of the method. That produces the same result when there are other await calls (the redis calls).

EDIT 1: like this

async pollQ() {
      // here we await other async functions (redis library calls)
      setTimeout(this.pollQ.bind(this), 100 + 100 * Math.random());
  }

I've tried using a driver function in the hope that when the pollQ scope is done, things will be gc-ed. But this had the same issue.

const driver = (() => {
    this.pollQ().finally(() => {
      setTimeout(driver, 100 + 100 * Math.random())
    })
  }).bind(this)

  driver()

Only doing a setInterval(pollQ, <inteval>) works without the heap growing incessantly but that's not ideal given the process might not complete in the interval. I'm out of ideas at this point and more importantly I don't understand the leak. The code doesn't look like it would create a leak but the heap keeps growing. Any help is appreciated.

EDIT 2:

It is part of a larger Nestjs application. The complete pollQ method is this:

async pollQ() {
    let message: RedisMessage;
    while (true) {
        try {
            message = await this.redisQueue.receiveMessage();
            if (message) {
                await this.process(message);
                await this.redisQueue.deleteMessage(message.id);
            }
        } catch (e) {
            this.logger.error(`Error polling redis q: ${e}`);
            if ((message._attempts || 0) < 3) {
                await this.redisQueue.returnToQueue(message.id);
            } else {
                await this.redisQueue.deleteMessage(message.id);
            }
        } finally {
            await new Promise((resolve) =>
                setTimeout(resolve, 100 + 100 * Math.random()),
            );
        }
    }
}

redisQueue.receiveMessage() returns immidiately regardless of whether the queue has a message. pollQ is called from onModuleInit which is a Nestjs specific lifecycle hook method for di container managed objects. The container creates an application scoped instance of the enclosing class.

async onModuleInit() {
    this.pollQ();
  }

The process method picks some fields from the event and persist it in a table.

The application is running on nodejs v20.19.0.


Solution

  • EDIT 2:

    I finally have the full picture thanks to a colleague's hawk eyes. In the codebase, AsyncLocalStorage is really from the npm package async-local-storage here. This package uses node's async hooks to create a new context (a vanilla js object) on every async operation start and keeps a reference to the parent's storage via the parent async resource's execution context id. You can see this here. Because we do this on a loop infinitely, this keeps piling up, impossible for the GC to clean up.

    One way to get rid of the chaining would be to call scope() on every iteration.

    EDIT 1:

    I finally have half an answer why this was happening. In the codebase, AsyncLocalStorage is enabled. This creates a context for each async call and chains past contexts for consecutive calls. I can reproduce this behavior with this isolated snippet:

    import { enable } from 'async-local-storage';
    
    enable();
    
    async function main() {
        while (true) {
            await new Promise((resolve) =>
              setTimeout(resolve, 100 + 100 * Math.random()),
            );
        }
      }
    
    main()
    

    What I want to understand is why the contexts are being chained since this promises here get resolved before creating the next one.

    Previous answer:

    I have settled for a setInterval based approach that waits for the processing code to finish before running it again.

    const driver = (() => {
        if (!this.polling) this.pollQ();
      }).bind(this);
    
     setInterval(driver, 100 + 100 * Math.random());
    

    and

    async pollQ() {
        this.polling = true;
        try {
          // awaits
        } catch (e) {
          // awaits
        } finally {
          this.polling = false;
        }
      }
    

    This heap is stable with this. But I'd still like to know why the original code grows the heap non stop.