[SOLVED] jdk6 Problematic frame: # J java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V

jdk6 Problematic frame: # J java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V

Is there a way to resolve this kind of error report:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fc955e66998, pid=25851, tid=140467030525696
#
# JRE version: 6.0_37-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.12-b01 mixed mode linux-amd64 compressed     oops)
# Problematic frame:
# J  java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V

The crash occurs quite frequently (1-2 times per day in web server production), almost always with different problematic frame report.

Here are examples of some error reports:

# J  java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter()Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;
# J  java.util.LinkedHashMap.addEntry(ILjava/lang/Object;Ljava/lang/Object;I)V
# C  [libc.so.6+0x6bb34]
# C  [libgobject-2.0.so.0+0x2346f]  g_type_check_instance_is_a+0x43
# C  [libgobject-2.0.so.0+0x2346f]  g_type_check_instance_is_a+0x43
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x32d166]  CardTableRS::write_ref_field_gc_par(void*, oopDesc*)+0x26
# V  [libjvm.so+0x7a33e2]  ContiguousSpace::prepare_for_compaction(CompactPoint*)+0x242
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x76943b]  ReferenceProcessor::balance_queues(DiscoveredList*)+0x32b
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x32d166]  CardTableRS::write_ref_field_gc_par(void*, oopDesc*)+0x26
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x4d3360]
# V  [libjvm.so+0x76943b]  ReferenceProcessor::balance_queues(DiscoveredList*)+0x32b

The only thing that seems to trigger the crashes is high memory usage approx 30gb, even though that has not always been the case (there are some crashes at instants where gc log shows low memory usage). The crashes do not occur when running in -Xint mode, but that mode is so slow that it is not an option.

Seems to be difficult to make any simple 'reproduceable code' to reproduce the error that occurs in production environment of a complex app.

What to do? I did report a bunch of these at the Oracle crash site though ...

I do not suspect hardware memory problems because nothing else ever crashes except java. And there is no custom native jni code in the application.

Our vm parameters are -server -Xss4096k -Xms32255M -Xmx32255M -Xnoclassgc -XX:+UseNUMA -XX:MaxPermSize=512m -XX:+UseGCOverheadLimit -verbose:gc -Xmaxf1 -XX:+UseCompressedOops -XX:+DisableExplicitGC -XX:+AggressiveOpts -XX:+ScavengeBeforeFullGC -XX:CMSFullGCsBeforeCompaction=10 -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:GCTimeRatio=19 -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=500 -Xloggc:gc.log.

Solution

Upgraded to jdk7 Java(TM) SE Runtime Environment (build 1.7.0_09-b05) and haven't had any problems since; follwing vmargs:

-server -Xss4096k -XX:MaxPermSize=512m -Xms32255M -Xmx32255M -Xnoclassgc -XX:+UseNUMA -XX:+UseBiasedLocking -XX:+UseFastAccessorMethods -XX:ReservedCodeCacheSize=48m -XX:+UseStringCache -XX:+HeapDumpOnOutOfMemoryError -XX:+UseGCOverheadLimit -Duser.timezone=EET -Xmaxf1 -XX:+UseCompressedOops -XX:+DisableExplicitGC -XX:+AggressiveOpts -XX:CMSInitiatingOccupancyFraction=70 -XX:+ParallelRefProcEnabled -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=100 -XX:+UseG1GC -XX:GCPauseIntervalMillis=3000 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -Xloggc:gc.log