redpanda-connectbloblang

Bloblang Inner Map Each


I am trying to apply templating by using redpanda connect documentation to build sqs inputs with dynamic parameters. End goal is to build URLs like below.

https://sqs.eu-west-1.amazonaws.com/123456789012/tr-dev-pip-myevent1
https://sqs.eu-west-1.amazonaws.com/123456789012/tr-dev-pip-myevent2
https://sqs.eu-west-1.amazonaws.com/123456789012/de-dev-pip-myevent1
https://sqs.eu-west-1.amazonaws.com/123456789012/de-dev-pip-myevent2

The problem is I couldn't write nested map_each blocks because when I run linting, I get

aws_sqs_list.yaml(40,15) required: expected query, got: 
})

aws_sqs_list.yaml(1,1) parse mapping: line 13 char 5: required: expected query
   |
13 |   })
   |     ^---

Template

name: aws_sqs_list
type: input
summary: Generates a list of SQS inputs based on dynamic parameters.

fields:
  - name: region
    type: string
    description: The AWS region for the SQS queues.
    default: eu-west-1
  - name: account_id
    type: string
    description: The AWS account ID.
  - name: cluster_name
    type: string
    description: The cluster identifier part of the queue name.
  - name: countries
    type: string
    kind: list
    description: A list of country codes to generate queues for.
  - name: event_names
    type: string
    kind: list
    description: A list of event names to generate queues for.

mapping: |
  root.broker.inputs = this.countries.map_each(country -> {
    this.event_names.map_each(event_name -> {
      "aws_sqs": {
        "url": "https://sqs.%s.amazonaws.com/%s/%s-%s-pip-%s".format(
          this.region,
          this.account_id,
          this.cluster_name,
          country,
          event_name
        ),
        "region": this.region
      }
    })
  })

Could you tell me if there's any workaround or how can I write nested loops here?


Solution

  • Try this:

    name: aws_sqs_list
    type: input
    summary: Generates a list of SQS inputs based on dynamic parameters.
    
    fields:
      - name: region
        type: string
        description: The AWS region for the SQS queues.
        default: eu-west-1
      - name: account_id
        type: string
        description: The AWS account ID.
      - name: cluster_name
        type: string
        description: The cluster identifier part of the queue name.
      - name: countries
        type: string
        kind: list
        description: A list of country codes to generate queues for.
      - name: event_names
        type: string
        kind: list
        description: A list of event names to generate queues for.
    
    mapping: |
      #!blobl
    
      root.broker.inputs = this.countries.map_each(country -> this.event_names.map_each(
        event_name -> {
          "aws_sqs": {
            "url": "https://sqs.%s.amazonaws.com/%s/%s-%s-pip-%s".format(
              this.region,
              this.account_id,
              this.cluster_name,
              country,
              event_name
            ),
            "region": this.region
          }
        }
      )).flatten()
    
    tests:
      - name: aws_sqs_list test
        config:
          region: eu-west-1
          account_id: 123456789012
          cluster_name: my-cluster
          countries:
            - tr
            - de
          event_names:
            - myevent1
            - myevent2
        expected:
          broker:
            inputs:
              - aws_sqs:
                  url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-tr-pip-myevent1"
                  region: eu-west-1
              - aws_sqs:
                  url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-tr-pip-myevent2"
                  region: eu-west-1
              - aws_sqs:
                  url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-de-pip-myevent1"
                  region: eu-west-1
              - aws_sqs:
                  url: "https://sqs.eu-west-1.amazonaws.com/123456789012/my-cluster-de-pip-myevent2"
                  region: eu-west-1
    

    You can run the test with this command:

    $ redpanda-connect template lint ./tmp/templates/aws_sqs_list.tmpl.yaml