amazon-web-servicesaws-cdkaws-step-functionsjson-path-expression

Combining JsonPath listAt and stringAt to make a single array in AWS Steps/CDK


I am invoking an AWS Steps orchestration - and one of the contained steps is to invoke an ECS Fargate Task.

I currently pass some of the Steps' inputs as the "command" of the ECS Task using JsonPath.listAt. I would like to also add a single extra value to the command array - but I cannot work out the syntax as a construct within CDK.

Our step's input is:

{
  "indexes": [
    "s3://bucket/test1.bam",
    "s3://bucket/test2.bam"
  ],
  "reference": "hg38"
}

And the example of us passing the indexes as command arguments to the ECS Task is:

new EcsRunTask(this, "Job", {
      integrationPattern: IntegrationPattern.RUN_JOB,
      cluster: fargateCluster,
      taskDefinition: taskDefinition,
      containerOverrides: [
        {
          command: JsonPath.listAt("$.indexes"),

I want to also pass JsonPath.stringAt("$.reference") in as the first argument of the command but I cannot get the syntax correct.

JsonPath.listAt is the only JsonPath function that returns a string[] (needed by .command in typescript CDK).

JsonPath.array can be used to build arrays - but cannot take in inputs in the same way as listAt (as in it won't flatten out an existing array input).


command: [
  JsonPath.stringAt("$.reference"),
  ...JsonPath.listAt("$.indexes"),
]

fails with

Cannot use JsonPath fields in an array, they must be used in objects


command: JsonPath.array(
            JsonPath.stringAt("$.reference"),
            ...JsonPath.listAt("$.indexes")
          ) as any

fails with

Error: Resolution error: Resolution error: Resolution error: Found an encoded list token string in a scalar string context. Use 'Fn.select(0, list)' (not 'list[0]') to extract elements from token lists..


Solution

  • The least bad option is a two step process: merge then flatten.

    First, add a Pass State to merge the inputs. The state will output a nested array: [["hg38"], ["s3://bucket/test1.bam", "s3://bucket/test2.bam"]]:

    const merge = new sfn.Pass(this, "merge", {
      parameters: {
        merge: sfn.JsonPath.array(
          sfn.JsonPath.array(sfn.JsonPath.stringAt("$.reference")),
          sfn.JsonPath.stringAt("$.indexes"), // CDK does not accept listAt here; the array param is of type []string
        ),
      },
      resultPath: "$.merge",
    });
    

    Now your command can use JSONPath wildcard syntax to flatten the array into the desired shape of ["hg38", "s3://bucket/test1.bam", "s3://bucket/test2.bam"]:

    command: sfn.JsonPath.listAt("$.merge.merge[*][*]")
    

    Note that this roundabout approach is needed because of how Step Functions' intrinsic functions work. It's not a limitation imposed by the CDK.