This one is baffling. All of a sudden we've started experiencing a situation where all threads in a Java process reach a BLOCKED state and the process is completely frozen. There's no jvm.log or hs*.log file to point to what's going on.
A jstack dump lists many threads, all in BLOCKED state, some of them as baffling as this:
Thread 14766: (state = BLOCKED)
- java.lang.String.substring(int, int) @bci=62, line=1939 (Compiled frame)
- java.util.Formatter.parse(java.lang.String) @bci=73, line=2469 (Compiled frame)
- java.util.Formatter.format(java.util.Locale, java.lang.String, java.lang.Object[]) @bci=213, line=2438 (Compiled frame)
- java.util.Formatter.format(java.lang.String, java.lang.Object[]) @bci=7, line=2367 (Compiled frame)
- java.lang.String.format(java.lang.String, java.lang.Object[]) @bci=9, line=2769 (Compiled frame)
(cropped the rest)
Or this:
Thread 14794: (state = BLOCKED)
- java.lang.Throwable.toString() @bci=17, line=344 (Interpreted frame)
- java.lang.String.valueOf(java.lang.Object) @bci=10, line=2826 (Compiled frame)
- java.io.PrintWriter.println(java.lang.Object) @bci=1, line=710 (Interpreted frame)
- java.lang.Throwable.printStackTrace(java.io.PrintWriter) @bci=6, line=509 (Compiled frame)
- org.apache.log4j.DefaultThrowableRenderer.render(java.lang.Throwable) @bci=19, line=60 (Compiled frame)
- org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep() @bci=50, line=87 (Interpreted frame)
- org.apache.log4j.spi.LoggingEvent.getThrowableStrRep() @bci=13, line=413 (Compiled frame)
- org.apache.log4j.WriterAppender.subAppend(org.apache.log4j.spi.LoggingEvent) @bci=26, line=313 (Compiled frame)
- org.apache.log4j.RollingFileAppender.subAppend(org.apache.log4j.spi.LoggingEvent) @bci=2, line=276 (Interpreted frame)
- org.apache.log4j.WriterAppender.append(org.apache.log4j.spi.LoggingEvent) @bci=10, line=162 (Compiled frame)
- org.apache.log4j.AppenderSkeleton.doAppend(org.apache.log4j.spi.LoggingEvent) @bci=106, line=251 (Compiled frame)
- org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(org.apache.log4j.spi.LoggingEvent) @bci=41, line=66 (Compiled frame)
- org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=26, line=206 (Compiled frame)
- org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=391 (Interpreted frame)
- org.apache.log4j.Category.error(java.lang.Object, java.lang.Throwable) @bci=37, line=322 (Interpreted frame)
(cropped the rest)
I must admit that while I've tried to search for a cause online, I don't know where to begin looking for it.
This is happening under jdk1.6.0_31
Are they all blocked in string operations? Have you run out of heap perm gen space maybe (do a 'jstat -gc' on the hung process)