Connect to a gRPC service on a GKE cluster from Cloud Build

We’re using a managed Kubeflow Pipelines (KFP) instance, created with GCP’s AI Platform Pipelines, with a managed MySQL instance, created with Cloud SQL. We’re also using Cloud Build to build and run our KFP pipelines. We’d like to add a Cloud Build step that, after a KFP pipeline run finishes, runs a script that uses the MLMD API to query for that pipeline run’s metadata. We have an MLMD script that can successfully query for metadata when run manually on a GCP VM. The issue has been getting that script to run in Cloud Build.

The first approach was to create the mlmd.metadata_store.MetadataStore object using MySQL connection information, e.g.,

connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.mysql.host = [IP address]
connection_config.mysql.port = 3306
connection_config.mysql.database = "kubeflow_experiments_metadata_metadata"
connection_config.mysql.user = [user]
connection_config.mysql.password = [password]
store = mlmd.metadata_store.MetadataStore(connection_config)

This runs fine on a VM. It appears, however, that the Cloud SQL Proxy is required for this to work in Cloud Build. Using this Cloud Functions codelab as an example, I was able to run a script in Cloud Build that used sqlalchemy to connect to MySQL via the proxy. The method for connecting sqlalchemy with the proxy, however, seems to be incompatible with the above MLMD API. It looks something like this:

driver_name = "mysql+pymysql"
query_string = dict({"unix_socket": "/cloudsql/{}".format(connection_name)})
db = sqlalchemy.create_engine(
    sqlalchemy.engine.url.URL(drivername=driver_name, username=[user], password=[password], database="kubeflow_experiments_metadata_metadata", query=query_string),
    pool_size=5,
    max_overflow=2,
    pool_timeout=30,
    pool_recycle=1800,
)

The second approach uses the MLMD gRPC service that’s deployed with KFP. First, I port forward the service:

kubectl port-forward svc/metadata-grpc-service 8080:8080

then create the mlmd.metadata_store.MetadataStore using the MLMD gRPC API:

connection_config = metadata_store_pb2.MetadataStoreClientConfig(
    host="127.0.0.1",
    port=8080,
)
store = mlmd.metadata_store.MetadataStore(connection_config)

Again, this works fine on a VM. I’m not sure, however, how to connect to the gRPC service from Cloud Build. My experience with gRPC and Kubernetes is limited, so I wouldn’t be surprised if there was a straightforward solution.

Any advice would be greatly appreciated!

Through other avenues, I was pointed to this article, which contains an example of how to port-forward KFP's ml-pipeline service in Cloud Build. I had to make one minor modification, which was to remove the "-n" and "kubeflow" arguments to the kubectl port-forward command. This specifies to kubectl to use the "kubeflow" namespace. GCP's AI Platform Pipelines, however, appears to create a "default" namespace when it deploys your KFP instance.