I have dockerized tomcat 7.0.73 based on java:7-jdk docker image.
I have a job that every 5 minutes does thread dump with command jstack 1. It works for about two weeks normally and after that crashes with message:
Unable to open socket file: target process not responding or HotSpot VM not loaded. The -F option can be used when the target process is not responding
Exit code is 1.
What can be the cause of this behavior?
JVM keeps working okay. Space looks correct too.
java -version
java version "1.7.0_111"
OpenJDK Runtime Environment (IcedTea 2.6.7) (7u111-2.6.7-2~deb8u1)
OpenJDK 64-Bit Server VM (build 24.111-b01, mixed mode)
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-202:1-xxx 10G 1013M 9.1G 10% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/xvda1 99G 68G 31G 69% /tmp
shm 64M 0 64M 0% /dev/shm
I suspect that a process may be clearing out old files in /tmp, and it's taking the relevant /tmp/.java_pid socket file with it (which is required by utilities such as jstack.) Next time it happens look to see if the relevant file is still there - if not, then that's your issue.