terraformterraform-provider-awsinfrastructureinfrastructure-as-code

Terraform plan --refresh-only return False Drift with empty tags in deployed resources


While using terraform plan --refresh-only to detect drifted resources, the command returns a plan where 1 or more resources are changed externally. All of them mark the tags attribute with an empty map {}

  # aws_iam_role.lalala_poweruser has changed
  ~ resource "aws_iam_role" "lalala_poweruser" {
        id                    = "lalala_poweruser"
        name                  = "lalala_poweruser"
      + tags                  = {}
        # (11 unchanged attributes hidden)
    }

To make emphasis, the + tags = {} shouldn't be marked with a + symbol as no one changed this property in the deployed infrastructure.

What I've tried so far:

The endgame of detecting drift using this command is to programmatically detect any change in a CICD pipeline before deploying new changes. If there is a real drift, we want to know, and we want to stop this deployment. Also, something helpful about this command is when a real drift is detected, it returns the diff so we can log the information and take action.

More info about this process from the Terraform docs:

Manage Resource Drift


Solution

  • In Terraform's architecture, one of the responsibilities for a provider is to read the current state of each remote object already under management, to compare it with the result of the most recent apply, and produce a new object which reconciles the two.

    Part of that responsibility is to deal with situations where values might appear different in Terraform but are effectively the same as far as the remote system is concerned. For each detected change, the provider must broadly place it into one of the following two categories:

    In the case you've shown here, it seems like this provider is not making a correct classification: there is no functional difference between tags = {} and omitting that argument altogether, and so the provider should've classified this as normalization and preserved the fact that you didn't set it, rather than reporting this as drift.

    You mentioned that you've already tried setting tags = {} explicitly in your configuration but that didn't fix the problem. Unfortunately that suggests a second opposite bug in the provider: during its create/update step it seems to be normalizing an empty map to be the same as null and so the provider disagrees with itself about what is the normalized form for this argument. If that is true then I don't think there will be anything you can do within your Terraform module to avoid the problem; the only recourse will be to fix the bug in the provider codebase.

    The AWS provider team is tracking various bugs of this type. That issue says that provider release v3.70.0 fixed a number of these bugs and so I would suggest making sure you're running at least that version as a first step. However there is a comment describing a similar problem with aws_iam_policy, and these two resource types are closely related and so probably share the same bugs in this regard. If you are already running the latest version of the provider then I think the best thing to do here is to report this by adding another comment to that same GitHub issue, so the provider team will be aware of it.

    I'm sorry I don't have an answer that can be implemented only within your Terraform module.