[SOLVED] Enable web access or interactive shell for PipelineJob tasks for Vertex AI

Enable web access or interactive shell for PipelineJob tasks for Vertex AI

I am trying to debug a PipelineJob that I launch on Vertex AI. Is there a way to enable web access on the components like you can when you launch Custom Jobs? This way, I could ssh into the running task and do a bunch of debugging.

Here is a simplified version of my pipeline code:

import kfp.v2.dsl as dsl

from google.cloud import aiplatform
from kfp.v2 import compiler
from kfp.v2.dsl import (
    component,
    Input,
    Output,
    Dataset,
    Metrics,
    Model,
    Artifact,
    graph_component
)
from copy import copy
from kfp.v2.google.client import AIPlatformClient
from typing import Optional, Dict, Union, List

@component(
    packages_to_install=['google-cloud-aiplatform']
)
def hello_world():
    import time
    print("Hello world")
    time.sleep(300)

@dsl.pipeline(
    name = "dataprep"
)
def train_model_pipeline(style: int):
    # Set Up Training and Test Data
    hello_op = hello_world()

I expected to be able to set enable_web_access(True) on the task, but that doesn't seem like an option because it's part of the CustomJob spec and not the PipelineTask.

Solution

As of now, there is no feature or any workaround to enable web access to kfp components.

There is already a filed feature request regarding this issue filed by @Optimus.

Posting the answer as community wiki for the benefit of the community that might encounter this use case in the future.

Feel free to edit this answer for additional information.