amazon-web-servicesaws-batchaws-step-functions

AWS Step Function - passing Input to container override for AWS batch job


I'm submitting an AWS Batch job from a Step Function. The batch job takes command line parameters. I can run it in the step function like this:

{
  "Comment": "Submit aws batch job with parameters",
  "StartAt": "SubmitJob",
  "States": {
    "SubmitJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName": "my-job",
        "JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:1",
        "JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
        "ContainerOverrides": {
          "Command": [
            "Hello",
            "World"
          ]
        }
      },
      "End": true
    }
  }
}

However, I would like to use the input to this step as the command line parameters. So I try to use the $.parameter notation in the Command.

My input is

{
  "param_1": "Hello",
  "param_2": "World"
}

and my step function is

{
  "Comment": "Submit aws batch job with parameters",
  "StartAt": "SubmitJob",
  "States": {
    "SubmitJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobName": "my-job",
        "JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:1",
        "JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
        "ContainerOverrides": {
          "Command": [
            "$.param_1",
            "$.param_2"
          ]
        }
      },
      "End": true
    }
  }
}

However, this results in my batch job receiving the raw strings like "$.param_1" rather than them being substituted by the input "Hello". Why? What is required instead?


Solution

  • This is one Solution/Workaraound, I'm not sure if it is the only option.

    In the AWS Batch Job Definition, in the Container properties, set Command to be

    ["Ref::param_1","Ref::param_2"]
    

    These "Ref::" links will capture parameters that are provided when the Job is run. Additionally, you can specify parameters in the job definition Parameters section but this is only necessary if you want to provide defaults.

    In the Step Functions State Machine definition, instead of using ContainerOverrides, use Parameters. Overall it will look like:

    {
      "Comment": "Submit aws batch job with parameters",
      "StartAt": "SubmitJob",
      "States": {
        "SubmitJob": {
          "Type": "Task",
          "Resource": "arn:aws:states:::batch:submitJob.sync",
          "Parameters": {
            "JobName": "my-job",
            "JobDefinition": "arn:aws:batch:us-west-1:123456789000:job-definition/my-job-definition:2",
            "JobQueue": "arn:aws:batch:us-west-1:123456789000:job-queue/my-job-queue",
            "Parameters": {
              "param_1.$": "$.param_1",
              "param_2.$": "$.param_2"
            }
          },
          "End": true
        }
      }
    }
    

    Thus the input into the step function gets passed to the Batch job which captures them and uses as Command.

    reference: https://docs.aws.amazon.com/batch/latest/userguide/example-job-definitions.html