I'm developing an Argo Workflow that requires passing params from step to step. In most cases, I am outputting the params into a file, which are then used in the input of the subsequent steps. These are ok and working as expected. The trouble I am having is with a new initial step that I'd like to implement. In this initial step, I am determining some parameters based on the original input, saving them to a file, and finally using (read: trying to use) the outputs in subsequent env declarations. Simplified version below:
templates:
### STEPS ORDER
- name: cua-pipeline-workflow
steps:
### STEP 1
- - name: setup-env
template: setup-env
### STEP 2
- - name: parser
template: parser-ingest
### STEP 1
- name: setup-env
script:
image: python:3.8-slim
command: [python]
source: |
import json
from pathlib import Path
output_dir = Path('/tmp')
Path(output_dir).mkdir(parents=True, exist_ok=True)
with open(output_dir.joinpath('env.txt'), 'w') as f:
f.write(json.dumps({'client': 'abc', 'pipeline': '123'}))
outputs:
parameters:
- name: envs
valueFrom:
path: '/tmp/env.txt'
#### STEP 2
- name: parser-ingest
container:
command: ["echo", "$CLIENT_NAME"]
image: python:3.8-slim
env:
- name: CLIENT_NAME
value: "{{ steps.setup-env.outputs.parameters.envs.client }}"
- name: PIPELINE
value: "{{ steps.setup-env.outputs.parameters.envs.pipeline }}"
In each attempt, the print statement looks like: "{{ steps.setup-env.outputs.parameters.envs.pipeline }}".
I have tried passing the output as an input to the next step then referencing the input in the env declaration. Single quotes, double quotes, no quotes around the steps output reference.
Is this functionality possible?
I found a solution here. It's a bit more verbose than I would have hoped, and a little bit of a compromise, but the solution works.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: param-flow-
labels:
workflows.argoproj.io/creator: argogod
spec:
serviceAccountName: workflows-service
entrypoint: param-flow
templates:
- name: param-flow
inputs:
parameters: null
steps:
- - name: setup-env
template: setup-env
- - name: say-something
template: say-something
arguments:
parameters:
- name: config
value: "{{ steps.setup-env.outputs.parameters.envs }}"
- name: setup-env
script:
image: python:3.8-slim
command: [python]
source: |
import json
from pathlib import Path
output_dir = Path('/tmp')
Path(output_dir).mkdir(parents=True, exist_ok=True)
with open(output_dir.joinpath('env.txt'), 'w') as f:
f.write(json.dumps({'client': 'abc', 'pipeline': '123'}))
outputs:
parameters:
- name: envs
valueFrom:
path: '/tmp/env.txt'
- name: say-something
inputs:
parameters:
- name: config
container:
image: python:3.8-slim
command: [sh, '-c']
args: ["echo {config: $CONFIG}"]
env:
- name: CONFIG
value: "{{ inputs.parameters.config }}"