google-cloud-platformgoogle-cloud-aigoogle-cloud-vertex-ai

Tracking resources used by VertexAI pipeline


Is it possible to track the resources consumed by a VertexAI pipeline run, similar to how it is possible to do for Dataflow where it shows a live graph of how many nodes are currently running to execute the pipeline?


Solution

  • Vertex AI Pipeline provides a feature for Visualizing and analyzing the pipeline results.

    This feature can be used to check the resource utilization once the Pipeline is deployed.

    steps:

    Go to vertex AI pipeline->
             Select a pipeline->
                   pipeline step->
                         view job(from Pipeline run analysis pane)
    

    In the View Job pane we can check for the resources utilized i.e machine types,machine count,CPU utilization graph for the pipeline step and we can view the logs too.

    enter image description here

    Utilizations:

    enter image description here

    As per this document, metrics from the Vertex AI like CPU utilization, CPU load are in the Beta launch stage. However, you can examine the metrics like CPU utilization from Cloud Monitoring by referring to this document and also find the below snap for more reference.

    enter image description here

    For changing the timeline of the graph you have to select the custom option in metrics explorer and provide the date and time for the duration that you want to view as shown in the below screenshot.

    enter image description here