amazon-web-servicesamazon-dynamodbamazon-cloudwatchrulesaws-iot-core

AWS DynamoDB not inserting all items from IoT Core with Rules engine

So I have a problem with an IoT solution I have created using AWS IoT Core, AWS Rules Engine and AWS DynamoDB.

The solution looks like the following: MQTT messages are published to a topic "topic/+/data" where the "+" is the thing that is publishing a message. There is a rule that intercepts these messages and is supposed to save the published message as a record in a DynamoDB table. The problem is that not all messages are persisted in the database. The DynamoDB tables capacity mode is set to be "On-demand".

Possible things that can have gone wrong:

  1. Not all messages are published to the topic stream
  2. The rule doesn't catch all messages
  3. DynamoDB isn't persisting all messages as it should

1 and 2 can be eliminated since AWS Cloudwatch IoT logs only shows successful "RuleExecution" events that correspond to the amount of messages published i.e if there was 600 messages published then there was 600 successful "RuleExecution" events. This was also tested for 500 messages published. The messages originated from a load test by Apache JMeter where different numbers of devices where simulated to post a message each second. The number of devices ranged from 10 to 100 and ran between 30 seconds and 15 minutes. All tests resulted in a large percentage, up to 30%, of messages that were not inserted in to the DynamoDB table even though Cloudwatch claims that they were successful insertions.

Somehow DynamoDB doesn't insert all the messages that are intercepted byt the rule even though Cloudwatch confirms that all the "RuleExecution" events are successful. Cloudwatch does not present any failed events either.

What could be the reason for this?

Solution

  • Collisions of the primary key were the problem. By using the timestamp() as the primary key and the traceid() as the sort key, everything works as intended. I can see now in the table that several entries have the same timestamp.

    To be clear: The rule used to insert messages in to DynamoDB (The IAM role is cropped out):

    Image