pythondockergoogle-cloud-vertex-aikubeflow-pipelines

Switch directory in a docker container within Vertex


I have a DBT docker container.

The WORKDIR is set to dbt/client/client_name.

I am easily able to run commands within the docker container using kfp.components.load_component_from_text()

However, somewhere along the line I need to switch directories to run other commands, the directory I am attempting to switch to is dbt/client/client_name_raw. What command can I use within my kfp.components.load_component_from_text() in order to switch to that directory?

I have already tried the following:

    @staticmethod
    def run_seed(client_name_raw, dbt_image_uri, target):
        run_seed = kfp.components.load_component_from_text(f'''
           name: DBT Client Seed
           implementation:
                container:
                  image: {dbt_image_uri}
                  command: ["cd" , "../{client_name_raw}", "dbt", "seed", "--target", "{target}"]
                ''')()
        return run_seed

Vertex gives me the following error message

"The replica workerpool0-0 exited with a non-zero status of 128. Termination reason: StartError. 
 Termination log message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "cd": executable file not found in $PATH: unknown
To find out more about why your job exited please check the logs:"

Seems like Vertex/kfp doesn't know how to interpret "cd" in this context. Any advice would be appreciated


Solution

  • I ended up finding the solution.

    I did it like this

        @staticmethod
        def run_seed(client_name_raw, dbt_image_uri, target):
            run_seed = kfp.components.load_component_from_text(f'''
               name: DBT Client Seed
               implementation:
                    container:
                      image: {dbt_image_uri}
                      command: ["bash", "-c", "cd ../{client_name_raw} && dbt deps && dbt seed --target {target}"]
                    ''')()
            return run_seed
    

    Hope it helps someone who encounters the same issue in future