gpukubectl

how to list all the pods who request special resource in k8s like gpu


GPU is limited resource in my k8s clusters, and I want to locate those pods easily in whole cluster.

$ kubectl get pods -A -o jsonpath="{.items[*].spec['containers'][*].resources.requests}"
{"cpu":"2","memory":"20Gi","nvidia.com/gpu":"1"} {"cpu":"10m","memory":"40Mi"} {"cpu":"1","memory":"2Gi"} 

How can I find out the pod list for those resource, for this case nvidia.com/gpu==1

I expect to have output like

POD     NAMESPACE     GPU RESOURCE
aab     abc           1
aab     abb           2

I guess it can be --field-selector or sed/awk


Solution

  • I think you can't do this using kubectl's JSONPath because there's no mechanism to capture e.g. metadata values (Pod name|namespace) from the down-scoped filtered value.

    You could use any number of tools but, my preference for processing JSON is jq and it has the advantage that, once you learn e.g. jq, you can use it everywhere you need to process JSON.

    I don't have access to a cluster but I spoofed the JSON output (hopefully correctly) and you should be able to use:

    FILTER=... (value below)

    ["Pod","Namespace","Container","GPU"],
    (
      .items[]
      |.metadata as $metadata
      |.spec.containers[]
      |.resources.requests as $requests
      |select($requests|has("nvidia.com/gpu"))
      |[
        $metadata.name,
        $metadata.namespace,
        .name,
        $requests["nvidia.com/gpu"]
       ]
    )|@tsv
    
    kubectl get pods \
    --all-namespaces \
    --output=json \
    | jq -r "${FILTER}"
    
    Explanation
    1. We output an array of arrays:
      • the first array is some row headers: ["Pod",...]
      • the second array is the output of the remaining filter
    2. We bind items[].metadata to the variable $metadata
    3. We scope to items[].spec.containers[]
    4. We then bind .resources.requests to $request
    5. We filter $requests by only those containing nvidia.com/gpu
    6. We then output an array comprising:
      • the $metadata variable's .name and .namespace values
      • the current (.) scope's name (i.e. the container's name)
      • the $requests variable's value of nvidia.com/gpu
    7. Finally @tsv turns an array into a table (tab delimited)

    Here's the jq play snippet