[SOLVED] The java application feign death. Jstack can make application recovery?

The java application feign death. Jstack can make application recovery?

Today i found that my application could not be accessed.Then i login in my server.I found that the application's thread was normal,But the CPU load was too high.

Then,i wanted to use jstack command.But it pointed out need -F option.And now i found the GC log stay in [GC (Allocation Failure) 2015-10-10T10:18:10.564+0800: 71147.518: [ParNew:, and there is no other logs.

Then,i used jstack -F command.Strange things happened.The CPU became normal,and my server also became normal,and The GC log also nomal,and the first line was [GC (Allocation Failure) 2015-10-10T10:17:50.757+0800: 53501.137: [ParNew: 210022K->245K(235968K), 369.6907808 secs] 400188K->1 90410K(1022400K), 369.6909604 secs] [Times: user=3475.15 sys=11.69, real=369.63 secs].And the jstack's result's runnable thread were all GC thread,like "Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f5f10021800 nid=0x6477 runnable

Here are my JAVA_OPTIONS

-server -Dfile.encoding=UTF-8 \ -Xms2g -Xmx2g -Xmn512m \ -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 \ -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75 \ -XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:/data/logs/gc-skynetlog_web.log \ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M \ -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/logs/dump_skynetlog_web.hprof

Solution

There is a fresh bug affecting Linux kernels 3.14 - 3.18

Symptoms, thread is consuming 100% while blocking on synchronization primitive. Sending signal (like jstack -F does) is fixing this situation. In case of JVM, if GC thread is affect by this, application will stay in STW pause forever.

You can find more details in this post by Gil Tene