[SOLVED] ImagePullBackOff on kubeflow

ImagePullBackOff on kubeflow

When I tried to run kubeflow pipelines on minikube with components whose Docker images are stored in Google Container Registry, I got ImagePullBackOff error.
I already set credential according to Ref1, by running a such following command.

kubectl create secret -n [my-namespace] generic user-gcp-sa \
  --from-file=user-gcp-sa.json=application_default_credentials.json

And I use use_gcp_secret method when defining a pipeline to use the registered credential according to Ref2.

@kfp.dsl.pipeline(name='example_pipeline')
def example_pipeline():
    example_component1_task = example_component1_op(
        input_1='/app/input.txt',
        input_2=8,
    ).apply(use_gcp_secret(secret_name='user-gcp-sa'))
    
    example_component2_task = example_component2_op(
        input_1=example_component1_task.outputs['output_1'],
        input_2=5,
    ).apply(use_gcp_secret(secret_name='user-gcp-sa'))

For building Docker images, I followed Ref3.

But after running pipeline by create_run_from_pipeline_func method and see Runs>Events section on Kubeflow dashboard, there is a message ImagePullBackOff.

I think Ref4 is irrelevant but I also tried minikube addons configure registry-creds but nothing changed as expected.

There is no other clues other than the ImagePullBackOff message, so I ask the question here.

Update

I found the message but I already tried all listed in Ref5. Also I added enough role to service account and created json key from it.

message: >-
      Failed to pull image
      "asia.gcr.io/myproject-123456/example_component1@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx":
      rpc error: code = Unknown desc = Error response from daemon: unauthorized:
      You don't have the needed permissions to perform this operation, and you
      may have invalid credentials. To authenticate your request, follow the
      steps in:
      https://cloud.google.com/container-registry/docs/advanced-authentication

If lack of information to answer the question, please tell me.

References

Solution

This could not be a perfect solution but I solved the issue by doing 2 things:

avoid to use "latest" tag
load docker images on minikube

minikube image load image_name:tag_name  
minikube image load image_name:tag_name