multithreadinggrailsweblogicmsgpackmessagepack

MessagePack template lookup causing stuck threads in WebLogic?


We're using MessagePack 0.6.6 for Java in Grails 2.0 on WebLogic 11g (10.3) to serialize string data...

public void serialize(Object object, OutputStream outputStream) 
   throws IOException {
   byte[] bytes = MessagePack.pack(object);
   outputStream.write(bytes);
   outputStream.flush();
}

The problem we're seeing in WebLogic is lots of STUCK threads, so we dumped the thread stack and found some threads getting stuck at org.msgpack.template.TemplateRegistry.lookup(TemplateRegistry:198), see dump below. We're confident our code did not introduce this issue since, in the example above, it's clear we're using MessagePack.pack() in a thread-safe manner. Looking at TemplateRegistry.java, line 198, lookup() is synchronized, but we're not sure why it's causing stuck threads.

        "[STUCK] ExecuteThread: 
    '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=43 idx=0xec tid=60 prio=1 alive, in native, blocked, daemon

-- Blocked trying to get lock: org/msgpack/template/TemplateRegistry@0xfffffffe8c2fb8e8[fat lock]
    at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
    at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1679)[optimized]
    at jrockit/vm/Locks.lockFat(Locks.java:1780)[optimized]
    at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1312)[optimized]
    at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]
    at jrockit/vm/Locks.monitorEnter(Locks.java:2466)[inlined]
    at jrockit/vm/Locks.monitorEnterForced(Locks.java:859)[optimized]
    at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)   
    at jrockit/vm/Locks.monitorEnterUnmatched(Ljava/lang/Object;)V(Native Method)
    at org/msgpack/template/TemplateRegistry.lookup(TemplateRegistry.java:198)[optimized]         
    at org/msgpack/MessagePack.write(MessagePack.java:195)[inlined]
    at org/msgpack/MessagePack.pack(MessagePack.java:639)[inlined]

Solution

  • You should use packer and unpacker as this blog says.

    MessagePack msgpack = new MessagePack(); // singleton
    Packer packer = msgpack.createPacker(outputStream); // createPacker every time
    packer.write(object);
    

    Following code may yield a perm gen memory error if you do too many since new MessagePack() loads classes every time.

    MessagePack msgpack = new MessagePack();
    byte[] bytes = msgpack.write(object);