google-kubernetes-engineminikubekubeflow

How to pass arguments when creating kubeflow pipeline yaml file?


Environment

I want to create kubeflow pipelines file to run pipeline on GKE.
I created the pipeline according to Getting started building a pipeline.
I cannot get kubeflow pipeline yaml file by following method.

kfp.compiler.Compiler().compile(
    pipeline_func=my_pipeline,
    package_path='pipeline.yaml',
    pipeline_parameters={'url': 'https://storage.googleapis.com/ml-pipeline-playground/iris-csv-files.tar.gz'})

I got TypeError: compile() got an unexpected keyword argument 'pipeline_parameters'
From this implementation, I should be able to use pipeline_parameters argument for kfp.compiler.Compiler().compile.

Reference


Solution

  • If you want to pass default pipeline parameters for your pipeline. You can do it directly in the function definition of the pipeline like this:

    @dsl.pipeline(name='name')
    def my_pipeline(a=1, b='default value'):
      # pipeline code
      pass
    

    For a minimal example of a full pipeline, you can take a look at this. From my experience working with Kubeflow, a lot of the documentation are wrong or not up-to-date. Many times, you have to find a workaround.