AWS Step Functions States.DataLimitExceeded Error

I am currently creating a Step Functions workflow which contains some Lambdas and other services that share states between one task and other.

Mainly I am doing some external requests on my Lambda and sharing between tasks to process this data.

When I am trying to test my Step Functions workflow I am receiving the following error: States.DataLimitExceeded, but when I test on the Lambda console everything works properly.

Error:

Can anyone help me to understand what I am doing wrong in Step Functions workflow?

The Step Functions service has a limit for the inputs/payloads of 256KB (more necessarily 262,144 bytes) of data as a UTF-8 encoded string. This quota affects tasks (activity, Lambda function, or integrated service), state or execution output, and input data when scheduling a task, entering a state, or starting an execution.

In your case looks like you are sending more than this limit between states, this is not a problem for the Lambda functions because the Lambda limit is 6MB for events.

To work with this, you need will to share these huge payloads in another service like S3 and then send this reference between your functions/states.

Also, if you do not need all the data that is being passed between states you can transform or filter the output using Filter output with OutputPath or Transform result with ResultSelector options and share only the required data that you will need for the next states.

For example, if you decide to use S3 for this case we can do something like this:

-> Lambda do the request 
-> Save on S3 the data 
-> Send the response with the S3 object ARN 
-> Get this file on the Lambda function and process the data

This is preventing you to send the payload directly in your Step functions and you will not need to worry with this 256Kb input limit.

If you want to transform or filter the output for your state you can do something like this:

This example shows how to transform your output data to get only the property you need in a object and filter a array to get only the first index in a array (you do not necessary need both).

To understand more about the Step Functions quotas/limits take a look here: Step Functions - Quotas