I've got a node.js server that is freezing in production, and it appears to be caused by an infinite loop inside of JSONStream. Here's a stack trace captured from a core dump from a frozen server:
1: toString [buffer.js:~392] (this=0x1e28fb6d25c9 <a Buffer>#1#,encoding=0x266ee104121 <undefined>,start=0x266ee104121 <undefined>,end=0x266ee104121 <undefined>)
2: arguments adaptor frame: 0->3
3: write [/home/deploy/node_modules/JSONStream/node_modules/jsonparse/jsonparse.js:136] (this=0x32cc8dd5a999 <a Parser>#2#,buffer=0x32cc8dd5aa49 <a Buffer>#3#)
4: /* anonymous */ [/home/deploy/node_modules/JSONStream/index.js:~17] (this=0x32cc8dd5ab11 <a Stream>#4#,chunk=0x32cc8dd5aa49 <a Buffer>#3#)
5: write [/home/deploy/node_modules/JSONStream/node_modules/through/index.js:~24] (this=0x32cc8dd5ab11 <a Stream>#4#,data=0x32cc8dd5aa49 <a Buffer>#3#)
6: write [_stream_readable.js:~582] (this=0x266ee106c91 <JS Global Object>#5#,dest=0x32cc8dd5ab11 <a Stream>#4#,i=0,list=0x266ee104101 <null>)
7: flow [_stream_readable.js:592] (this=0x266ee106c91 <JS Global Object>#5#,src=0x32cc8dd5ac69 <an IncomingMessage>#6#)
8: /* anonymous */ [_stream_readable.js:560] (this=0x266ee106c91 <JS Global Object>#5#)
9: _tickCallback [node.js:415] (this=0x29e7331bb2a1 <a process>#7#)
How can I find the source of this infinite loop?
Unfortunately the servers are running in production and are processing many thousands of requests, so it's difficult to give any additional context. The basic function of the servers is to make outbound HTTP requests for other services.
It's worth noting that I don't believe this is caused by a memory leak. The server's memory usage remains constant (and low) during these freeze events, while the CPU spikes to 99%
Another piece of evidence towards the conclusion of an infinite loop is that the event loop itself seems to have stopped. When I put a console.log inside of a setInterval, the server will stop outputting as soon as it freezes.
We have verified that the problem is not caused by expired/corrupted socket connections, by setting the number of max connections to Infinity (which disables their reuse in node.js)
We're using JSONStream 0.7.1 (which includes a default jsonparse version of 0.0.5). We found this issue on the JSONStream repo, and tried forking JSONParse and only updating to the newest jsonparse version. It did NOT fix the issue.
It would appear that your issue is somehow being caused by this line of jsonstream@0.0.5. While I can't conclusively fix the issue for you without a dump of what is going through the server, this would appear to indicate that your buffer is extremely large.
This would also explain why your servers are locking up (as you mentioned in chat), why the event loop doesn't proceed and why your memory doesn't go up to the sky but your CPU does; what's probably happen here is you're attempting to toString() an incredibly large amount of bytes which your hardware simply can't do and it dies.
By all means report back on any further investigation from this lead. I feel it's unavoidable that you're going to have repro the issue on a dev box. It could be as simple as adding some sanity checks to your buffer and making sure it doesn't go above a certain size.
The above section of code is only hit if n <= 128, so if you're using signed characters (such as Unicode) or signed bytes then you might actually encounter this issue in normal function... might be something to consider!