javaspring-bootkubernetesamazon-eksjava-11

At any given point how to Collect HeapDump for the running program from a pod


I've got Springboot apps which uses OpenJDK running in container and runs as pid 1. These are non root containers and imx remote enabled. I am trying to generate heap dumps on demand for these running pods.

I figured for that I need to use jmap to write to the location but it seems per this , it always tries to write to /tmp which is readonly. How Can I overcome this and allow JVM to write a on demand heapdump to custom location or fix this ?

At the moment it seems it's hardcoded to /tmp why ?.

https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/jdk.attach/aix/classes/sun/tools/attach/VirtualMachineImpl.java#L40-L45

what options do I have to generate heapdump for these pods on demand ?

Thanks


Solution

  • Regarding readonly issue.

    If the container is running with readOnlyFilesystem: true, you need to exclude /tmp directory by providing an emptyDir on that path.

    First, add a volume such as:

    volumes:
    - name: tmp
      emptyDir: {}
    

    Then, mount it:

    volumeMounts:
    - name: tmp
      mountPath: /tmp
    

    And now apps inside the container can write in /tmp directory.

    Regarding dump

    jmap should support setting the dump location, example:

    jmap -dump:live,format=b,file=/the/desired/path/dump.hprof 1
    

    I think your issue is that the attach code expects java to be able to write something under /tmp, maybe the pid of the process and more, so it's not related to where you write the dump file, but it's more related to java not being able to create tmp files needed during the process.