We're seeing an interesting (though quite severe) issue with one of our application servers: at a certain point in time, the CPU usage of the JVM running our web applications starts rising and keeps on rising until the applications eventually slow down to a crawl. The only way to fix it is to restart the Application server software.
This brings me to a simple question; what can we do to troubleshoot this?
I have considered using VisualVM (or some other JVM monitoring tool), but the best they can do - in this particular case - is give me a thread dump, which still will not tell me what is eating up all the CPU time (unless I'm missing something).
You need to find out what it is doing? A common cause of this problem is running low on free memory. If this is not the cause, a CPU profiler is needed. VisualVM comes free with the JDK and can do both for you.
only you can't profile the application all the time
When this is happening you can do an ad hoc profiling by calling jstack multiple times a few second apart. You can use diff of the stack traces to help you can find the thread which are likely to be busy and consuming CPU.