I am trying to apply templating by using redpanda connect documentation to build sqs inputs with dynamic parameters. End goal is to build URLs like below.
https://sqs.eu-west-1.amazonaws.com/123456789012/tr-dev-pip-myevent1
https://sqs.eu-west-1.amazonaws.com/123456789012/tr-dev-pip-myevent2
https://sqs.eu-west-1.amazonaws.com/123456789012/de-dev-pip-myevent1
https://sqs.eu-west-1.amazonaws.com/123456789012/de-dev-pip-myevent2
The problem is I couldn't write nested map_each
blocks because when I run linting, I get
aws_sqs_list.yaml(40,15) required: expected query, got:
})
aws_sqs_list.yaml(1,1) parse mapping: line 13 char 5: required: expected query
|
13 | })
| ^---
name: aws_sqs_list
type: input
summary: Generates a list of SQS inputs based on dynamic parameters.
fields:
- name: region
type: string
description: The AWS region for the SQS queues.
default: eu-west-1
- name: account_id
type: string
description: The AWS account ID.
- name: cluster_name
type: string
description: The cluster identifier part of the queue name.
- name: countries
type: string
kind: list
description: A list of country codes to generate queues for.
- name: event_names
type: string
kind: list
description: A list of event names to generate queues for.
mapping: |
root.broker.inputs = this.countries.map_each(country -> {
this.event_names.map_each(event_name -> {
"aws_sqs": {
"url": "https://sqs.%s.amazonaws.com/%s/%s-%s-pip-%s".format(
this.region,
this.account_id,
this.cluster_name,
country,
event_name
),
"region": this.region
}
})
})
Could you tell me if there's any workaround or how can I write nested loops here?
Try this:
name: aws_sqs_list
type: input
summary: Generates a list of SQS inputs based on dynamic parameters.
fields:
- name: region
type: string
description: The AWS region for the SQS queues.
default: eu-west-1
- name: account_id
type: string
description: The AWS account ID.
- name: cluster_name
type: string
description: The cluster identifier part of the queue name.
- name: countries
type: string
kind: list
description: A list of country codes to generate queues for.
- name: event_names
type: string
kind: list
description: A list of event names to generate queues for.
mapping: |
#!blobl
root.broker.inputs = this.countries.map_each(country -> this.event_names.map_each(
event_name -> {
"aws_sqs": {
"url": "https://sqs.%s.amazonaws.com/%s/%s-%s-pip-%s".format(
this.region,
this.account_id,
this.cluster_name,
country,
event_name
),
"region": this.region
}
}
)).flatten()
tests:
- name: aws_sqs_list test
config:
region: eu-west-1
account_id: 123456789012
cluster_name: my-cluster
countries:
- tr
- de
event_names:
- myevent1
- myevent2
expected:
broker:
inputs:
- aws_sqs:
url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-tr-pip-myevent1"
region: eu-west-1
- aws_sqs:
url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-tr-pip-myevent2"
region: eu-west-1
- aws_sqs:
url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-de-pip-myevent1"
region: eu-west-1
- aws_sqs:
url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-de-pip-myevent2"
region: eu-west-1
You can run the test with this command:
$ redpanda-connect template lint ./tmp/templates/aws_sqs_list.tmpl.yaml