I have an IoT workflow with various devices sending data to an S3 bucket, which triggers a Lambda. The Lambda processes the data and may trigger an alert (e.g. due to DeviceX having a temperature above Y). I wish to receive email alerts when a new device encounters such an alert with a maximum of receiving one email per device per day.
The solution I am hoping to implement is to setup the SNS writer to output an email body for the SNS Topic endpoint which includes the device ID (which is globally unique across my devices), the alert ID and the date.
In the SNS setup I want to make it so that my subscribers receive an email when the first such event happens. But since the Lambda may be triggered several times per hour by the same device ID, they should not receive a new email for every such invocation, even if the alert condition is still true. If another device encounters an alert, that should still trigger a new separate email - and similarly, if the first device continues to be in the alert state the next day, a new email will be sent because now the date changes.
I believe this may be linked to subscription filters or deduplication, but I have not been able to find the right solution for this.
The SNS message filtering feature is not helpful for this use case. The SNS deduplication feature only works within a 5 minute time window. You can't count on SNS to handle this for you if you care about a 24 hour time period. You need to have code in your Lambda function that determines if it actually needs to send the message to SNS or if the same message has already been sent recently.
The best way to handle this is to create a DynamoDB table that your Lambda function stores your message in before it sends them. Before sending a message, it would check if that message already exists in the DynamoDB table, if so, it would skip sending the message to SNS.
You would configure TTL expiration on the DynamoDB messages, so that they are automatically deleted from the table after 24 hours.
One thing to be aware of, the record in DynamoDB may not actually be deleted until some time after the TTL value expires. So for the solution to be more precise, your Lambda function would need to check if the message already exists in the database, and if it does exist, compare the record's TTL value against the current time before deciding if it should skip sending the message again.