terraform

With Terraform how to convert map of strings to map of objects


I have a Terraform variables.tf in the module root with contents similar to this:

variable "parameters" {
  description = "Parameter Store key/values"
  type = map(string)
  default = {
    "/customer1/prod/keycloak/password" = "password123"
    "/customer1/prod/keycloak/realm" = "default"
    "/customer1/prod/keycloak/url" = "https://customer1-sso.app.something.cloud/auth"
    "/customer1/prod/keycloak/userid" = "f:48c1ce1f:monitoring"
    "/customer1/uat/keycloak/password" = "password123"
    "/customer1/uat/keycloak/realm" = "uat"
    "/customer1/uat/keycloak/url" = "https://customer1-sso.npr.app.something.cloud/auth"
    "/customer1/uat/keycloak/userid" = "f:d48d4452:monitoring"
    "/customer1/uat2/keycloak/password" = "password123"
    "/customer1/uat2/keycloak/realm" = "uat2"
    "/customer1/uat2/keycloak/url" = "https://customer1-sso.npr.app.something.cloud/auth"
    "/customer1/uat2/keycloak/userid" = "f:5fe762fd:monitoring"
    "/customer2/prod/keycloak/password" = "password123"
    "/customer2/prod/keycloak/realm" = "default"
    "/customer2/prod/keycloak/url" = "https://customer2-sso.app.something.cloud/auth"
    "/customer2/prod/keycloak/userid" = "f:a053a488:monitoring"
    "/customer2/uat/keycloak/password" = "password123"
    "/customer2/uat/keycloak/realm" = "uat"
    "/customer2/uat/keycloak/url" = "https://customer2-sso.npr.app.something.cloud/auth"
    "/customer2/uat/keycloak/userid" = "f:225118c6:monitoring"
    "/customer2/uat2/keycloak/password" = "password123"
    "/customer2/uat2/keycloak/realm" = "uat2"
    "/customer2/uat2/keycloak/url" = "https://customer2-sso.npr.app.something.cloud/auth"
    "/customer2/uat2/keycloak/userid" = "f:9d8b2d30:monitoring
 }
}

I would like to convert it into a map of objects that looks like this:

{
  "customer1_prod" = {
    customer = "customer1"
    password = "password123"
    realm    = "default"
    url      = "https://customer1-sso.app.something.cloud/auth"
    userid   = "f:48c1ce1f:monitoring"
  }
  ...
}

I have tried this in the root module (main.tf) code:

# Transform flat map of strings into a grouped map of customer-env -> { password, realm, url, userid }
locals {
  # Step 1: Flatten parameters to tuples of [group_key, key_in_object, value]
  parameter_tuples = [
    for full_key, value in var.parameters : {
      group_key = join("_", slice(split("/", full_key), 1, 3))     # e.g. customer1_prod
      key       = split("/", full_key)[3]                          # e.g. password
      value     = value
    }
  ]

  # Step 2: Group by environment and assemble objects with customer field
  customers = {
    for group_key in distinct([for p in local.parameter_tuples : p.group_key]) :
    group_key => merge(
      {
        customer = split("_", group_key)[0]
      },
      merge([
        for p in local.parameter_tuples : {
          for inner in [p] :
          inner.key => inner.value
        } if p.group_key == group_key
      ]...)
    )
  }
}

output "all_customers" {
  value = local.customers
}

module "dd_synthetic_browser" {
  source = "./modules/dd_synthetic_browser"
  customers = local.customers
}

However, the output shows this:

+ all_customers             = {
      + customer1_prod = {
          + customer = "customer1"
          + keycloak = "f:48c1ce1f:monitoring"
        }
    ...
    ...

How to fix this incorrect map of objects? There are two errors:

  1. there should be five items in the object (customer, password, realm, url and userid) and there are only two.
  2. the keycloak = "f:48c1ce1f:monitoring" item should be userid = "f:48c1ce1f:monitoring"

Also, I am open to suggestions of better Terraform code to perform this conversion because I think my code is too complicated for its task and could probably be accomplished in a simpler fashion with less lines of code.


Solution

  • There's an off by one error

    Let's reduce the volume of inputs to the minimum to investigate:

    variable "parameters" {
      description = "Parameter Store key/values"
      type = map(string)
      default = {
        "/customer1/prod/keycloak/password" = "password123"
        "/customer1/prod/keycloak/realm" = "default"
        "/customer1/prod/keycloak/url" = "https://customer1-sso.app.something.cloud/auth"
        "/customer1/prod/keycloak/userid" = "f:48c1ce1f:monitoring"
     }
    }
    

    And take a look at the intermediary variable parameter_tuples:

    locals {
      parameter_tuples = [
        for full_key, value in var.parameters : {
          group_key = join("_", slice(split("/", full_key), 1, 3))     # e.g. customer1_prod
          key       = split("/", full_key)[3]                          # e.g. password
          value     = value
        }
      ]
    }
    
    output "parameter_tuples" {
      value = local.parameter_tuples
    }
    

    We get the following:

    $ terraform plan
    
    Changes to Outputs:
      + parameter_tuples = [
          + {
              + group_key = "customer1_prod"
              + key       = "keycloak"
              + value     = "password123"
            },
          + {
              + group_key = "customer1_prod"
              + key       = "keycloak"
              + value     = "default"
            },
          + {
              + group_key = "customer1_prod"
              + key       = "keycloak"
              + value     = "https://customer1-sso.app.something.cloud/auth"
            },
          + {
              + group_key = "customer1_prod"
              + key       = "keycloak"
              + value     = "f:48c1ce1f:monitoring"
            },
        ]
    

    The key is 'keycloak' for all tuples because split("/", full_key), passed a string like /customer1/prod/keycloak/realm, will return data of the form:

    [
        "",
        "customer1",
        "prod",
        "keycloak", # <- 3
        "realm",
    ]
    

    Fixed up

    So just bumping that index:

    locals {
      parameter_tuples = [
        for full_key, value in var.parameters : {
          group_key = join("_", slice(split("/", full_key), 1, 3))     # e.g. customer1_prod
          key       = split("/", full_key)[4]                          # 3->4
          value     = value
        }
      ]
      customers = {
        for group_key in distinct([for p in local.parameter_tuples : p.group_key]) :
        group_key => merge(
          {
            customer = split("_", group_key)[0]
          },
          merge([
            for p in local.parameter_tuples : {
              for inner in [p] :
              inner.key => inner.value
            } if p.group_key == group_key
          ]...)
        )
      }
    }
    
    output "all_customers" {
      value = local.customers
    }
    
    

    produces the desired output:

    $ terraform plan
    
    Changes to Outputs:
      + all_customers = {
          + customer1_prod = {
              + customer = "customer1"
              + password = "password123"
              + realm    = "default"
              + url      = "https://customer1-sso.app.something.cloud/auth"
              + userid   = "f:48c1ce1f:monitoring"
            }
        }