In AWS-SDK for Node.js getParametersByPath() SSM method returns paged data, meaning I have to make several calls in a loop. Is there a way to disable pagination or increase page size (beyond 10, which is max for MaxNumberparameter) ?
Alternatively, if several calls have to be made - can results be collected into array of Promises to be resolved at once?
Basically, we have about 12 parameters that we need to load on a Lambda startup. Currently getting parameters takes about 2 seconds - because 2 consecutive calls are made. Ideally it has to either be one call or 2 calls made in parallel.
This is a good question. Let me outline some concepts, AWS design decisions and how I would work around the limitations they impose.
All AWS client SDKs, including the AWS-SDK for Node.js, make API calls towards the respective AWS service's REST API, e.g. the API action GetParametersByPath of the AWS Systems Manager endpoint [1].
The AWS APIs usually implement the so called cursor-based pagination concept. This type of pagination has other characteristics than page-based pagination (i.e. classic offset-limit pagination) - described in several Medium articles [2][3].
One major limitation of cursor-based pagination is: It doesn’t allow sending parallel requests for different batches. There is no way to jump for a specific page, it needs to iterate through all the prior pages.
Looking at the pros and cons of each pagination type, the AWS engineers probably made a trade-off in favor of consistency at the cost of performance.
AWS states that the valid range for the MaxResults parameter is 1-10. That means, there is no way for a client to extend the page size beyond 10. All SDKs must adhere to that limit when implementing pagination. Disabling pagination [4] has the only effect of returning a single page of up to 10 items.
You could sync the SSM API results to a custom DynamoDB table and query DynamoDB instead of SSM. If this solution is viable depends on several characteristics, e.g. consistency requirements, how often SSM parameters are changing and confidentiality of SSM parameter values (or which parameter attributes are read by the Lambda function).
You could send concurrent requests to the API if you split the set of expected results properly and use the ParameterFilters API parameter [1]. You can split by following keys: tag:.+|Name|Type|KeyId|Path|Label|Tier|DataType using the BeginsWith or Equals option. [5]
This solution requires that you can make assumptions about the results you expect to receive. You must ensure that a particular subset does not contain more than 10 entries. If there are e.g. max. 10 SSM parameters per department, you could do the following: Send one request via getParametersByPath() per department, specifying MaxResults=10 and Path=/department/<name> or ParameterFilters=[{Key=tag:department,Option=Equals,Values=["<name>"]}]. These department-specific requests could be sent concurrently.
Workaround 2 could be considered the "two concurrent requests" approach you mentioned in the question and workaround 1 is the "one single request" approach. Workarounds require to either drop the consistency property or making assumptions about the data partitioning.
[1] https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetParametersByPath.html#systemsmanager-GetParametersByPath-request-NextToken
[2] https://medium.com/innomizetech/how-to-build-an-api-with-aws-lambda-to-paginate-data-using-serverless-41c4b6b676a4
[3] https://medium.com/swlh/paginating-requests-in-apis-d4883d4c1c4c
[4] https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html#cli-usage-pagination-serverside
[5] https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_ParameterStringFilter.html#systemsmanager-Type-ParameterStringFilter-Option