pythonargo-workflowsargo

Passing output parameters to env declaration in ArgoWF


I'm developing an Argo Workflow that requires passing params from step to step. In most cases, I am outputting the params into a file, which are then used in the input of the subsequent steps. These are ok and working as expected. The trouble I am having is with a new initial step that I'd like to implement. In this initial step, I am determining some parameters based on the original input, saving them to a file, and finally using (read: trying to use) the outputs in subsequent env declarations. Simplified version below:

templates:
   ### STEPS ORDER
  - name: cua-pipeline-workflow
    steps:
      ### STEP 1
      - - name: setup-env
          template: setup-env
     ### STEP 2
      - - name: parser
          template: parser-ingest
### STEP 1
- name: setup-env
  script:
    image: python:3.8-slim
    command: [python]
    source: |
        import json
        from pathlib import Path
        output_dir = Path('/tmp')
        Path(output_dir).mkdir(parents=True, exist_ok=True)
        with open(output_dir.joinpath('env.txt'), 'w') as f:
            f.write(json.dumps({'client': 'abc', 'pipeline': '123'}))
  outputs:
    parameters:
      - name: envs
        valueFrom:
          path: '/tmp/env.txt'
#### STEP 2
- name: parser-ingest
  container:
    command: ["echo", "$CLIENT_NAME"]
    image: python:3.8-slim
    env:
      - name: CLIENT_NAME
        value: "{{ steps.setup-env.outputs.parameters.envs.client }}"
      - name: PIPELINE
        value: "{{ steps.setup-env.outputs.parameters.envs.pipeline }}"

In each attempt, the print statement looks like: "{{ steps.setup-env.outputs.parameters.envs.pipeline }}".

I have tried passing the output as an input to the next step then referencing the input in the env declaration. Single quotes, double quotes, no quotes around the steps output reference.

Is this functionality possible?


Solution

  • I found a solution here. It's a bit more verbose than I would have hoped, and a little bit of a compromise, but the solution works.

    apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    metadata:
      generateName: param-flow-
      labels:
        workflows.argoproj.io/creator: argogod
    spec:
      serviceAccountName: workflows-service
      entrypoint: param-flow
      templates:
        - name: param-flow
          inputs:
            parameters: null
          steps:
            - - name: setup-env
                template: setup-env
            - - name: say-something
                template: say-something
                arguments:
                  parameters:
                    - name: config
                      value: "{{ steps.setup-env.outputs.parameters.envs }}"
        - name: setup-env
          script:
            image: python:3.8-slim
            command: [python]
            source: |
                import json
                from pathlib import Path
                output_dir = Path('/tmp')
                Path(output_dir).mkdir(parents=True, exist_ok=True)
                with open(output_dir.joinpath('env.txt'), 'w') as f:
                    f.write(json.dumps({'client': 'abc', 'pipeline': '123'}))
          outputs:
            parameters:
              - name: envs
                valueFrom:
                  path: '/tmp/env.txt'
        - name: say-something
          inputs:
            parameters:
              - name: config
          container:
            image: python:3.8-slim
            command: [sh, '-c']
            args: ["echo {config: $CONFIG}"]
            env:
              - name: CONFIG
                value: "{{ inputs.parameters.config }}"