I have a DBT docker container.
The WORKDIR is set to dbt/client/client_name
.
I am easily able to run commands within the docker container using kfp.components.load_component_from_text()
However, somewhere along the line I need to switch directories to run other commands, the directory I am attempting to switch to is dbt/client/client_name_raw
. What command can I use within my kfp.components.load_component_from_text()
in order to switch to that directory?
I have already tried the following:
@staticmethod
def run_seed(client_name_raw, dbt_image_uri, target):
run_seed = kfp.components.load_component_from_text(f'''
name: DBT Client Seed
implementation:
container:
image: {dbt_image_uri}
command: ["cd" , "../{client_name_raw}", "dbt", "seed", "--target", "{target}"]
''')()
return run_seed
Vertex gives me the following error message
"The replica workerpool0-0 exited with a non-zero status of 128. Termination reason: StartError.
Termination log message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "cd": executable file not found in $PATH: unknown
To find out more about why your job exited please check the logs:"
Seems like Vertex/kfp doesn't know how to interpret "cd" in this context. Any advice would be appreciated
I ended up finding the solution.
I did it like this
@staticmethod
def run_seed(client_name_raw, dbt_image_uri, target):
run_seed = kfp.components.load_component_from_text(f'''
name: DBT Client Seed
implementation:
container:
image: {dbt_image_uri}
command: ["bash", "-c", "cd ../{client_name_raw} && dbt deps && dbt seed --target {target}"]
''')()
return run_seed
Hope it helps someone who encounters the same issue in future