I'm using Tomcat 9 on a Centos 7 VPS. This server has run fine for two years, but recently crashes as if it's out of memory. The crash occurs between 1 hour and 24 hours after fresh Tomcat kill and start. I have also restarted my VPS. I removed all app logs, to be sure some limit wasn't being reached. There are no errors in any logs, and I've read through all similar issues going back to 2010.
The Eclipse MemoryAnalyzer seemed hopeful, so I added the HeapDumpOnOutOfMemoryError flag to my setenv.sh as shown:
export JAVA_OPTS="-Xms1500m -Xmx1500m -Xss512m -Xmn1000m -XX:+HeapDumpOnOutOfMemoryError";
echo "setenv.sh has set JAVA_OPTS: $JAVA_OPTS";
I also increased all memory parameters by 500m. But, no heap file was found in the JVM directory, or any other directory.
I am the only user of the site. And between crashes, I restore the main page as proof that TC is still alive, until the eventual crash.
TC hosts static content, and two WAR files. The crashes still occur whether I remove one, or both, of the WARs. I did update my static pages by changing the copyright year to 2021 (something I usually do in January, but forgot this year). I have pulled the original static files from backup, and it still crashes.
Can anyone offer some other ideas on how to track this down?
In the comments it looked like there's a solution, but I'm sceptical about it:
I've recommended to switch from JAVA_OPTS to CATALINA_OPTS (here's the reason)- but that was just a side note.
As much as it makes sense, I wouldn't expect it to fix anything: Your Tomcat ends up running with exactly the same configuration, just from a different environment variable.
The only way where I could imagine it fixing anything (specifically memory related) is when both JAVA_OPTS and CATALINA_OPTS declare non-matching memory parameters and somehow the lower one from both wins. In the end, Tomcat is started with both parameters combined and the JVM will only interpret one of them (don't ask me if it's the first, last or undefined)
My recommended general approach is still in the comments: Start by ruling out one assumption after the other - monitor the memory consumption and garbage collection if you assume that you have a memory problem. There it'll be obvious if you expect 8GB heapspace, but you're running on 1GB. Or a memory leak will be obvious. Once you can rule out memory (because the crash happens when there's plenty of memory available) go after the next assumption.
Not quite concrete... I'm wondering if this answer actually helps someone in hindsight...