kubernetesprometheusmonitoringvictoriametrics

Kubernetes pod custom monitoring


I'm trying to implement python script which collect and parsing kubernetes pod manifest's image version and secretName of each pod in 2 various kubernetes clusters and then if there are any differences between 2 clusters - should be send an alert. These metrics for 2 clusters then should be parsed by instance of Victoria Metrics. The problem is observed in if i check kubectl describe pod_name - in its output exists field secretName:

Volumes:
  cacert:
    Type:           Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:  glusterfs-cluster
    Path:           test/jvm/cert
    ReadOnly:       false
  service-conf-secrets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          example-app-1.25.01-57409t3
    SecretOptionalName:  <nil>

But if I use kubernetes.client.CoreV1Api and its function list_pod_for_all_namespaces - can't find in its output secretName at all.

Where can I find and parse this field and make prometheus format metrics from these fields?


Solution

  • Here's an example.

    I've includes comment references to the Python SDK's implementation of the Kubernetes types as well as the type hints for these types to help explain the use of the properties.

    I've included the full enumeration of V1VolumeProjection names including secret (V1SecretProjection) for completeness.

    from kubernetes import client,config
    
    config.load_kube_config()
    
    v1 = client.CoreV1Api()
    
    # https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1PodList.md
    pod_list: client.V1PodList = v1.list_pod_for_all_namespaces(watch=False)
    # Iterator over returned items (if any)
    pods: list[client.V1Pod] = pod_list.items
    for pod in pods:
        metadata: client.V1ObjectMeta = pod.metadata
        # https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1PodSpec.md
        spec: client.V1PodSpec = pod.spec
        print(f"{metadata.namespace}/{metadata.name} [{pod.status.pod_ip}]")
        # if pod.metadata.namespace=="shell-operator" and pod.metadata.name=="pods":
        # Iterative over volume (f any)
        # https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Volume.md
        volumes: list[client.V1Volume] = spec.volumes
        for volume in volumes:
            if volume.projected:
                projected: client.V1ProjectedVolumeSource = volume.projected
                # https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1VolumeProjection.md
                sources: list[client.V1VolumeProjection] = projected.sources
                for source in sources:
                    # if source.config_map:
                    # if source.downward_api:
                    if source.secret:
                        # https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1SecretProjection.md
                        secret: client.V1SecretProjection = source.secret
                        print(secret.name)
                        items: list[client.V1KeyToPath] = secret.items
                        for i in items:
                            path: str = i.path
                            print(path)
                    # if source.service_account_token: