terraformamazon-cloudwatchamazon-snspagerduty

Pagerduty AWS Integration How to Subscribe to a Non-Alert Based SNS Topic


I have read where CloudWatch Alerts can be used to trigger PagerDuty alerts as well as auto resolve the PD alert when the CloudWatch Alert backs down to an OK state.

It seems that SNS topics that are based on CloudWatch Alerts have a payload with a message{} member that PagerDuty expects for this type of integration.

https://support.pagerduty.com/main/docs/amazon-cloudwatch-integration-guide#why-are-my-cloudwatch-events-not-triggering-incidents-in-pagerduty

I would like to bind an SNS Topic Subscription to a PagerDuty CloudWatch API Integration that is not based on a CloudWatch Alert, but rather just anything else writing to that topic in the message property.

For example, DMS will write messages to SNS Topics (not CloudWatch alert based) when Replication Tasks fail. When a PagerDuty Subscription is created for these DMS SNS Topics, the message is never received.

Is there a way to force this or do the messages have to be based on CloudWatch Alerts for the expected payload?

The only alternative that I can think of is to subscribe the DMS SNS Topics to a Lambda and create a fake CloudWatch Alert payload and send those to another SNS topic that is has subscriptions with PagerDuty endpoints.

Any better ideas, can I deliver non alert based SNS topics to PagerDuty hassle free?

Terraform for setup

locals{
  pagerduty_alerts_endpoint = "https://events.pagerduty.com/integration/xxxx/enqueue"
}

resource "aws_sns_topic" "dms_repl_tasks_topic_pagerduty" {

    name     = "dms-repl-tasks-topic-pagerduty-${local.namespace_coalesce}"  
    delivery_policy = local.sns_delivery_policy_pagerduty
    tags     = merge(local.common_tags, local.migrated_tags, tomap({ "Name" = "dms-repl-tasks-topic-pagerduty-${local.namespace_coalesce}" }))

    depends_on = [
      aws_dms_replication_instance.dms_instance_on_prem_to_rds
    ]
}

resource "aws_sns_topic_subscription" "repl_task_notification_pagerduty_subscription" {
   count = "${local.pagerduty_alerts_endpoint != "" ? 1 : 0}"
   topic_arn = aws_sns_topic.dms_repl_tasks_topic_pagerduty.arn
   protocol    = "https"
   endpoint    = "${local.pagerduty_alerts_endpoint}"
   endpoint_auto_confirms = true

   depends_on = [
     aws_sns_topic.dms_repl_tasks_topic_pagerduty
   ]
}

resource "aws_dms_event_subscription" "event_subscription_repl_task_direct" {

enabled             = true
event_categories    = ["creation", "failure", "deletion", "state change", "configuration change"]
name                = "dms-event-sub-repl-task-direct-${local.namespace_coalesce}"
sns_topic_arn       = aws_sns_topic.dms_repl_tasks_topic.arn
source_ids          = [for ts in aws_dms_replication_task.onprem_to_rds_replication_task : ts.replication_task_id]
source_type         = "replication-task"
tags                = merge(local.common_tags, tomap({ "Name" = "dms-event-sub-repl-task-direct-${local.namespace_coalesce}" }))

depends_on = [
  aws_dms_replication_task.onprem_to_rds_replication_task,
  aws_sns_topic.dms_repl_tasks_topic,
  aws_sns_topic_subscription.repl_task_notification_email_subscription
]

timeouts {
    create = "2m"
    delete = "2m"  
    update = "2m"
   }
}   

EDIT: All of the above works seamlessly in PagerDuty. In this case the admin was not forwarding the message to the place where the integration was being validated.


Solution

  • The integration workings via SNS without Alert metadata. The PagerDuty admin did not configure the messages to be forwarded to the place where integration testing was taking place. :/

    enter image description here