I'm trying to test out creating a monitor for google pub sub and am getting an "Invalid Query" error. This is the query text when i view source of another working monitor, so i'm confused as to why this isn't working.
Error: Error: error creating monitor: 400 Bad Request: {"errors":["The value provided for parameter 'query' is invalid"]}
Terraform:
resource "datadog_monitor" "bad_stuff_sub_monitor" {
name = "${var.customer_name} Bad Stuff Monitor"
type = "metric alert"
message = "${var.customer_name} Bad Stuff Topic getting too big. Notify: ${var.datadog_monitor_notify_list}"
escalation_message = "Escalation message @pagerduty"
query = "avg:gcp.pubsub.subscription.num_undelivered_messages{project_id:terraform_gcp_test}"
thresholds = {
ok = 0
warning = 1
warning_recovery = 0
critical = 2
critical_recovery = 1
}
notify_no_data = false
renotify_interval = 1440
notify_audit = false
timeout_h = 60
include_tags = true
# ignore any changes in silenced value; using silenced is deprecated in favor of downtimes
lifecycle {
ignore_changes = [silenced]
}
tags = [var.customer_name, var.project_name]
}
So I ended up just looking at the tests in the datadog terraform provider and noticing the query format they are testing.
query = "avg(last_30m):avg:gcp.pubsub.subscription.num_undelivered_messages{project_id:${var.project_name},subscription_id:{project_id:terraform_gcp_test} > 2"
It seems you need to specify a time range and also add in a comparison threshold that matches your critical alert threshold. That was what was missing.