How to find out which thread holds the monitor?

My application is using Gson 2.2 for converting POJOs to JSON. When I was making a load test I stumbled upon a lot of threads blocked in Gson constructor:

"http-apr-28201-exec-28" #370 daemon prio=5 os_prio=0 tid=0x0000000001ee7800 nid=0x62cb waiting for monitor entry [0x00007fe64df9a000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at com.google.gson.Gson.<init>(Gson.java:200)
    at com.google.gson.Gson.<init>(Gson.java:179)

Thread dump does NOT show any threads holding [0x00007fe64df9a000] monitor. How can I find out who holds it?

Gson code at line 200 looks pretty innocent:

// built-in type adapters that cannot be overridden
factories.add(TypeAdapters.STRING_FACTORY);
factories.add(TypeAdapters.INTEGER_FACTORY);

I'm using JRE 1.8.0_91 on Linux

Solution

tl;dr I think you are running into GC-related behavior, where threads are being put in waiting state to allow for garbage collection.

I do not have the whole truth but I hope to provide some pieces of insight.

First thing to realize is that the number in brackets, [0x00007fe64df9a000], is not the address of a monitor. The number in brackets can be seen for all threads in a dump, even threads that are in running state. The number also does not change. Example from my test dump:

main" #1 prio=5 os_prio=0 tid=0x00007fe27c009000 nid=0x27e5c runnable [0x00007fe283bc2000]
   java.lang.Thread.State: RUNNABLE
        at Foo.main(Foo.java:12)

I am not sure what the number means, but this page hints that it is:

... the pointer to the Java VM internal thread structure. It is generally of no interest unless you are debugging a live Java VM or core file.

Although the format of the trace explained there is a bit different so I am not sure I am correct.

The way a dump looks when the address of the actual monitor is shown:

"qtp48612937-70" #70 prio=5 os_prio=0 tid=0x00007fbb845b4800 nid=0x133c waiting for monitor entry [0x00007fbad69e8000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233)
        - waiting to lock <0x00000005b8d68e90> (a java.lang.Object)

Notice the waiting to lock line in the trace and that the address of the monitor is different from the number in brackets.

The fact that we cannot see the address of the monitor involved indicates that the monitor exists only in native code.

Secondly, the Gson code involved does not contain any synchronization at all. The code just adds an element to an ArrayList (assuming no bytecode manipulation has been done and nothing fishy is being done at low level). I.e., it would not make sense to see the thread waiting for a standard synchronization monitor at this call.

I found some, indications that threads can be shown as waiting for a monitor entry when there is a lot of GC going on.

I wrote a simple test program to try to reproduce it by just adding a lot of elements to an array list:

List<String> l = new ArrayList<>();
while (true) {
    for (int i = 0; i < 100_100; i++) {
            l.add("" + i);
    }
    l = new ArrayList<>();
}

Then I took thread dumps of this program. Occasionally I ran into the following trace:

"main" #1 prio=5 os_prio=0 tid=0x00007f35a8009000 nid=0x12448 waiting on condition [0x00007f35ac335000]
   java.lang.Thread.State: RUNNABLE
      at Foo.main(Foo.java:10)   <--- Line of l.add()

While not identical to the OP's trace, it is interesting to have a thread waiting on condition when no synchronization is involved. I experienced it more frequently with a smaller heap, indicating that it might be GC related.

Another possibility could be that code that contains synchronization has been JIT compiled and that prevents you from seeing the actual address of the monitor. However, I think that is less likely since you experience it on ArrayList.add. If that is the case, I know of no way to find out the actual holder of the monitor.