azurekubernetesazure-aksheap-dumpthread-dump

Taking Thread dump/ Heap dump of Azure Kubernetes pods


We are running our kafka stream application on Azure kubernetes written in java. We are new to kubernetes. To debug an issue we want to take thread dump of the running pod.

Below are the steps we are following to take the dump.

  1. Building our application with below docker file.

    FROM mcr.microsoft.com/java/jdk:11-zulu-alpine
    RUN apk update && apk add --no-cache gcompat
    RUN addgroup -S user1 && adduser -S user1 -G user1
    USER user1
    WORKDIR .
    COPY target/my-application-1.0.0.0.jar .
    
  2. Submitting the image with below deployment yaml file

apiVersion: apps/v1 kind: Deployment metadata: name: my-application-v1.0.0.0 spec: replicas: 1 selector: matchLabels: name: my-application-pod app: my-application-app template: metadata: name: my-application-pod labels: name: my-application-pod app: my-application-app spec: nodeSelector: agentpool: agentpool1 containers: - name: my-application-0 image: myregistry.azurecr.io/my-application:v1.0.0.0 imagePullPolicy: Always command: ["java","-jar","my-application-1.0.0.0.jar","input1","$(connection_string)"] env: - name: connection_string valueFrom: configMapKeyRef: name: my-application-configmap key: connectionString resources: limits: cpu: "4" requests: cpu: "0.5"

  1. To get a shell to a Running container you can run the command below:

    kubectl exec -it <POD_NAME> -- sh

  2. To get thread dump running below command

    jstack PID > threadDump.tdump

but getting permission denied error enter image description here

Can some one suggest how to solve this or steps to take thread/heap dumps. Thanks in advance


Solution

  • Since you likely need the thread dump locally, you can bypass creating the file in the pod and just stream it directly to a file on your local computer:

    kubectl exec -i POD_NAME -- jstack 1 > threadDump.tdump
    

    If your thread dumps are large you may want to consider piping to pv first to get a nice progress bar.