Update: How do i add additional group or user asigned managed identity and use users/groups names instead of UUIDs?
Main.tf
provider "azurerm" {
features {}
}
locals {
access_map = {
owner_other_access = {
permissions_access = "---"
type = "other"
},
owner_group_access = {
permissions_access = "r-x"
type = "group"
},
owner_mask_access = {
permissions_access = "rwx"
type = "mask"
},
owner_user_access = {
permissions_access = "rwx"
type = "user"
}
}
default_map = {
owner_other_default = {
permissions_default = "---"
type = "other"
},
owner_group_default = {
permissions_default = "rwx"
type = "group"
},
owner_mask_default = {
permissions_default = "rwx"
type = "mask"
},
owner_user_default = {
permissions_default = "rwx"
type = "user"
}
}
}
resource "azurerm_storage_data_lake_gen2_filesystem" "this" {
for_each = var.storage_containers
name = each.value.sc_name
storage_account_id = each.value.storage_account_id
properties = {}
dynamic "ace" {
for_each = merge(local.access_map, jsondecode(each.value.acl_access))
iterator = item
content {
type = item.value.type
scope = "access"
permissions = item.value.permissions_access
id = lookup(item.value, "id", null)
}
}
dynamic "ace" {
for_each = merge(local.default_map, jsondecode(each.value.acl_default))
iterator = item
content {
type = item.value.type
scope = "default"
permissions = item.value.permissions_default
id = lookup(item.value, "id", null)
}
}
}
teraform.tfvars
storage_containers_fs = {
raw_fs_container = {
storage_account_id = "bronze_storage_account"
sc_name = "rawfs"
acl_access = "{\"owner_group_access\":{\"permissions_access\":\"r-x\",\"type\":\"group\",\"id\":\"3f97daxxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\",\"id\":\"f5509xxxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\"}}"
acl_default = "{\"owner_group_default\":{\"permissions_default\":\"rwx\",\"type\":\"group\",\"id\":\"3f97daxxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\",\"id\":\"f5509xxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\"}}"
}
}
varaibles.tf
variable "storage_containers_fs" {
description = "Storage Containers settings"
type = map(object({
storage_account_id = string
sc_name = string
acl_access = string
acl_default = string
}))
}
My problem with this code is: When deploying for the very first time, this code works and creates a private container with ACL inside, assigning the correct rights to the two identities, in this case two groups, as seen above
When deploying this code again with a new group added to thte existing ones, the code fails by trying to replace the first group in the list with the new one added.
The new tfvars will look like this:
storage_containers_fs = {
raw_fs_container = {
storage_account_id = "bronze_storage_account"
sc_name = "rawfs"
acl_access = "{\"owner_group_access\":{\"permissions_access\":\"r-x\",\"type\":\"group\",\"id\":\"3f97daxxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\",\"id\":\"f5509xxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\",\"id\":\"deb25xxxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\"}}"
acl_default = "{\"owner_group_default\":{\"permissions_default\":\"rwx\",\"type\":\"group\",\"id\":\"3f97daxxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\",\"id\":\"f5509xxxx-xxxxxxx-xxxxx-xxxxxx-xxxxxxxxx\"}}"
}
}
This is the output from apply:
Terraform will perform the following actions:
# module.az_bronze_storage_containers_fs.azurerm_storage_data_lake_gen2_filesystem.this["raw_fs_container"] will be updated in-place
~ resource "azurerm_storage_data_lake_gen2_filesystem" "this" {
id = "https://xxxxxxxxxxxxxxxxx.dfs.core.windows.net/rawfs"
name = "rawfs"
# (4 unchanged attributes hidden)
- ace {
- permissions = "---" -> null
- scope = "access" -> null
- type = "other" -> null
}
- ace {
- permissions = "---" -> null
- scope = "default" -> null
- type = "other" -> null
}
- ace {
- permissions = "rwx" -> null
- scope = "access" -> null
- type = "mask" -> null
}
- ace {
- permissions = "rwx" -> null
- scope = "access" -> null
- type = "user" -> null
}
- ace {
- permissions = "rwx" -> null
- scope = "default" -> null
- type = "mask" -> null
}
- ace {
- permissions = "rwx" -> null
- scope = "default" -> null
- type = "user" -> null
}
- ace {
- id = "3f97xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx" -> null
- permissions = "r-x" -> null
- scope = "access" -> null
- type = "group" -> null
}
+ ace {
+ id = "deb25xxx-xxxx-xxxx-xxxx-xxxxxxxxxxx"
+ permissions = "r-x"
+ scope = "access"
+ type = "group"
}
+ ace {
+ permissions = "---"
+ scope = "access"
+ type = "other"
}
+ ace {
+ permissions = "---"
+ scope = "default"
+ type = "other"
}
+ ace {
+ permissions = "rwx"
+ scope = "access"
+ type = "mask"
}
+ ace {
+ permissions = "rwx"
+ scope = "access"
+ type = "user"
}
+ ace {
+ permissions = "rwx"
+ scope = "default"
+ type = "mask"
}
+ ace {
+ permissions = "rwx"
+ scope = "default"
+ type = "user"
}
# (1 unchanged block hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
module.az_bronze_storage_containers_fs.azurerm_storage_data_lake_gen2_filesystem.this["raw_fs_container"]: Modifying... [id=https://xxxxxxxxxxxxxxxxx.dfs.core.windows.net/rawfs]
╷
│ Error: setting access control for root path in File System "rawfs" in Storage Account "xxxxxxxxxxxxxxxxx": datalakestore.Client#SetAccessControl: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationPermissionMismatch" Message="This request is not authorized to perform this operation using this permission.\nRequestId:e59de7e1-b01f-0066-41cd-3f5884000000\nTime:2024-01-05T11:48:36.0364932Z"
│
│ with module.az_bronze_storage_containers_fs.azurerm_storage_data_lake_gen2_filesystem.this["raw_fs_container"],
│ on ..\templates\az-storage-filesystem\main.tf line 42, in resource "azurerm_storage_data_lake_gen2_filesystem" "this":
│ 42: resource "azurerm_storage_data_lake_gen2_filesystem" "this" {
Not sure why this doesn't work.
Ideally is to add a new group without removing the existing one (first problem) AND can we make this template use group names instead of UUIDs in the tfvars (second problem)? Maybe using data.azuread_users, data.azuread_groups, data.azuread_serviceprincipals? Hardcoding IDs in the tfvars doesn't seem like a good idea. I would personally add the group name instead of the ID and let the template resolve the ID for me.
I was expecting a new identity to be added instead of replacing an existing one.
Solution
Ideally is to add a new group without removing the existing one (first problem).
To modify the existing container , you may need Storage Blob Data Owner role. Follow the Ms doc for more details
can we make this template use group names instead of
UUIDsin thetfvars(second problem)? Maybe usingdata.azuread_users,data.azuread_groups,data.azuread_serviceprincipals? Hardcoding IDs in the tfvars doesn't seem like a good idea.
If you create containers using .tfvars, it accepts an ID, not a name. However, you can use the code below to create a container without hardcoding the group ID by using a data block
terraform.tfvars
storage_containers = {
container1 = {
storage_account_id = ""
sc_name = "container1"
acl_access = "{\"owner_user_access\":{\"permissions_access\":\"rwx\",\"type\":\"user\"},\"owner_group_access\":{\"permissions_access\":\"r-x\",\"type\":\"group\",\"id\":\"data.azuread_group.example.id\"},\"owner_other_access\":{\"permissions_access\":\"---\",\"type\":\"other\"},\"owner_mask_access\":{\"permissions_access\":\"rwx\",\"type\":\"mask\"}}"
acl_default = "{\"owner_user_default\":{\"permissions_default\":\"rwx\",\"type\":\"user\"},\"owner_group_default\":{\"permissions_default\":\"rwx\",\"type\":\"group\",\"id\":\"data.azuread_group.example.id\"},\"owner_other_default\":{\"permissions_default\":\"---\",\"type\":\"other\"},\"owner_mask_default\":{\"permissions_default\":\"rwx\",\"type\":\"mask\"}}"
},
container2 = {
storage_account_id = ""
sc_name = "container2"
acl_access = "{\"owner_user_access\":{\"permissions_access\":\"rwx\",\"type\":\"user\"},\"owner_group_access\":{\"permissions_access\":\"r-x\",\"type\":\"group\",\"id\":\"Group-ID\"},\"owner_other_access\":{\"permissions_access\":\"---\",\"type\":\"other\"},\"owner_mask_access\":{\"permissions_access\":\"rwx\",\"type\":\"mask\"}}"
acl_default = "{\"owner_user_default\":{\"permissions_default\":\"rwx\",\"type\":\"user\"},\"owner_group_default\":{\"permissions_default\":\"rwx\",\"type\":\"group\",\"id\":\"Group-ID\"},\"owner_other_default\":{\"permissions_default\":\"---\",\"type\":\"other\"},\"owner_mask_default\":{\"permissions_default\":\"rwx\",\"type\":\"mask\"}}"
}
}
Here is the updated code to create an containers without .tfvars file.
provider "azurerm" {
features {}
}
locals {
access_map = {
container1 = {
permissions_access = "rwx"
type = "group"
},
container2 = {
permissions_access = "rwx"
type = "user"
}
}
default_map = {
group = {
permissions_default = "rwx"
type = "group"
},
user = {
permissions_default = "rwx"
type = "user"
},
}
}
data "azuread_group" "example" {
display_name = "Azure-AD-Group-Name"
}
data "azurerm_storage_account" "example" {
name = "storage-account-name"
resource_group_name = "RG-name"
}
resource "azurerm_storage_data_lake_gen2_filesystem" "example" {
for_each = local.access_map
name = each.key
storage_account_id = data.azurerm_storage_account.example.id
dynamic "ace" {
for_each = [data.azuread_group.example.id]
content {
id = ace.value
permissions = local.access_map[each.key].permissions_access
scope = "access"
type = local.access_map[each.key].type
}
}
dynamic "ace" {
for_each = local.default_map
content {
type = ace.key
scope = "default"
permissions = ace.value.permissions_default
id = data.azuread_group.example.id
}
}
}
Output:
- terraform tags for list variable not working
- Terraform 13, Validate variable base on the value of another
- How do I reference a resource created from different module in my current module?
- Terraform - GCP - Import project IAM member
- Creating an Azure Linux VM with Ubuntu 20.04 with Terraform
- Terraform doesn't allow to use variable when creating resource. How to go around?
- Authenticating using the Azure CLI is only supported as a User (not a Service Principal)
- Terraform outputs 'Error: Variables not allowed' when doing a plan
- Issue with Azure key vault and Azure storage account deployment
- temporary_name_for_rotation must be specified when changing any of the following properties: pod_subnet_id
- Why is TF build failing with "Error refreshing state: HTTP remote state endpoint requires auth"?
- With Gitlab, in the CI/CD pipeline, how can I dynamically create and save SSH key pairs as CI/CD project variables?
- Conditionally add element to existing map
- Parse HCL block into Golang map
- Run an AWS ECS task
- Terraform Error: Error locking state: Error acquiring the state lock: 2 errors occurred:
- Terraform: Error: retrieving static website properties for Storage Account (Subscription: *** : context deadline exceeded
- How to add additional security groups to EKS cluster?
- How to use a blue/green deployment style with CodeDeploy and Auto Scaling Groups that allows for repetitive deployments in Terraform?
- Self referenced Security group inbound rule in AWS RDS terraform?
- How to run this terraform nested for loop?
- Error: Failure when executing a Terraform Resource creation request
- Conditionally add property to object
- Registering a private endpoint in Azure private DNS automatically using Terraform
- How to deal with "Error: Duplicate object key" in Terraform
- Unable to use VARs from a tfvars file in Terraform
- Terraform AWS Provider: SecretsManager can't apply because version was deleted
- Experimenting Locally with Terraform
- Upgrade terraform to specific version
- When I used Terraform to organize Docker for deployment, pulling the image failed