google-cloud-vertex-aikubeflow-pipelines

How to set up a delay between two Vertex AI components in a pipeline?


I am working on a pipeline using Vertex AI, and I'm facing an issue with scheduling the execution of two components with a delay. Here's the scenario:

  1. Component A generates some data as its output.
  2. Component B processes the output from Component A and sends the result to an external provider for further processing. This external provider typically takes around 2 hours to complete its task.
  3. Component C should be scheduled to run 2 hours after Component B has finished.

My question is, how can I set up this delay between Component B and Component C in the pipeline?

I have considered using the pipeline orchestration capabilities provided by Vertex AI, but I couldn't find a built-in feature for adding such delays between components. I also looked into using Cloud Scheduler, but it seems more suitable for triggering pipeline runs rather than introducing delays within a pipeline.

I would greatly appreciate any insights or suggestions on how to accomplish this delay between the execution of Component B and Component C in the Vertex AI pipeline. Thank you in advance for your help!


Solution

  • I'd recommend having a separate pipeline for component C. When the component B in the first pipeline completes, publish a message to PuSub and then trigger the Pipeline of Component C with that message in PubSub. For more info: https://cloud.google.com/vertex-ai/docs/pipelines/trigger-pubsub