pythongoogle-cloud-platformgoogle-cloud-vertex-aimlopsvertex-ai-pipeline

Vertex AI automatic retraining


I’m trying to create a Vertex AI endpoint with Monitoring enabled that can trigger a Vertex AI pipeline execution when one of the deployed models drops its performance. However, Vertex AI does not provide any built-in feature to do it. Is there a method to capture the alert thrown by Vertex AI Monitoring and trigger the Pipeline?


Solution

    1. The Vertex AI Model Monitoring jobs are logged as part of Cloud Logging 1.
    2. You can react to those loggings using log-based alerts 2. For that, you need to configure a notification channel to PubSub 3
    3. Based on those PubSub message you can trigger a Cloud Function 4
    4. The Cloud Function can initiate the Vertex AI Pipeline run to re-train the model 5

    https://cloud.google.com/vertex-ai/docs/model-monitoring/using-model-monitoring#cloud-logging-info

    https://cloud.google.com/logging/docs/alerting/log-based-alerts

    https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.notificationChannels#NotificationChannel

    https://cloud.google.com/functions/docs/calling/pubsub

    https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.pipelineJobs/create