terraform destroys and recreates cloud run services on every alternating run, for no reason, with no change?

I have some cloud run services with the latest version of terraform (1.3.9) as well as older versions.

It seems that on every alternating run on my cloud run services, terraform will do this:

 ~ template {
          ~ metadata {
              ~ annotations = {
                  - "client.knative.dev/user-image"           = "gcr.io/yeo-dev/gateway:latest" -> null
                  - "run.googleapis.com/client-name"          = "gcloud" -> null
                  - "run.googleapis.com/client-version"       = "418.0.0" -> null
                    # (3 unchanged elements hidden)
                }
                name        = "gateway-00134-yek"
                # (2 unchanged attributes hidden)
            }

And on the next run with no changes, will reverse that exact change.

Why is this happening and how can I possibly prevent it?

Terraform cloud run definitions:

  template {
    metadata {

      annotations = {
        # Limit scale up to prevent any cost blow outs!
        "autoscaling.knative.dev/maxScale" = "5"
        # Use the VPC Connector
        "run.googleapis.com/vpc-access-connector" = google_vpc_access_connector.cloudrun.name
        # all egress from the service should go through the VPC Connector
        "run.googleapis.com/vpc-access-egress" = "private-ranges-only"
      }
    }

    spec {
      service_account_name = module.cloudrun-sa.email
      containers {
        image = "gcr.io/${var.gcp_project}/gateway:latest"

The client-name and client-version as well as the user image is never nulled.

Provider versions:

terraform {
  cloud {
    organization = "redacted"
    workspaces {
      tags = ["main"]
    }

  }

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 4.39.0"
    }
  }
}

I have 7 cloud run services in my terraform, this happens to all of them.

I've had ~4 separate terraform cloud projects, this was happening with all of them.

top level definitions:

  autogenerate_revision_name = true
  lifecycle {
    ignore_changes = [template[0].spec[0].containers[0].image, status[0].latest_created_revision_name, status[0].latest_ready_revision_name, template[0].metadata[0].name]
  }

I've added the above so I can run deployments using gcloud run deploy in my github actions/pipelines without breaking terraform.

---

EDIT:

Potential solution is to add those annotations to the ignore lifecycle:

Bringing the total ignore changes list to:

    ignore_changes = [template[0].metadata[0].annotations["run.googleapis.com/client-name"],
      template[0].metadata[0].annotations["client.knative.dev/user-image"],
      template[0].metadata[0].annotations["run.googleapis.com/client-version"],
      template[0].spec[0].containers[0].image,
      status[0].latest_created_revision_name,
      status[0].latest_ready_revision_name,
    template[0].metadata[0].name]

This seems to work for now, but I'm not sure on the implications with larger changes like taking down and rebuilding the whole project.

Posting this as a community wiki.

As per @SebastianG, a potential solution/workaround is to add annotations to the ignore lifecycle, bringing the total ignore list changes to:

    ignore_changes = [template[0].metadata[0].annotations["run.googleapis.com/client-name"],
      template[0].metadata[0].annotations["client.knative.dev/user-image"],
      template[0].metadata[0].annotations["run.googleapis.com/client-version"],
      template[0].spec[0].containers[0].image,
      status[0].latest_created_revision_name,
      status[0].latest_ready_revision_name,
    template[0].metadata[0].name]

In addition, another community member suggested to raise this as an issue by filing a bug through this link.