pythonamazon-web-servicesamazon-dynamodbboto3

How to update nested attribute name for specific DynamoDB items


I use boto3 client for managing DynamoDB items. Let's assume that the item structure is the following:

{
  "id": "item_id",
  "name": "item_name",
  "age": 23,
  "body": {
    {"nested": "nested_data"}
  }
}

There was a bug at some point, and instead of nested key, nested_error key was inserted for some items:

{
  "id": "item_id",
  "name": "item_name",
  "age": 23,
  "body": {
    {"nested_error": "nested_data"}
  }
}

The bug has been fixed and I would like to implement a "migration" that will rename nested_error key to nested one for all items affected. The action should be skipped if the name of the attribute is already nested.

What is the correct way to do such a migration? Should I implement a script with boto3 or an awscli command would be enough? Or maybe it can be done via the AWS Console? What command should I use?


Solution

  • The best approach depends on how much data you have. I would personally opt for the CLI, as it auto paginates. One thing to be careful of is throttling exceptions should it be a large data set. Might be easily fixed implementing some sleep in the code:

    #!/bin/bash
    
    TABLE_NAME="YourTableName"
    
    SCAN_RESULTS=$(aws dynamodb scan \
      --table-name $TABLE_NAME \
      --filter-expression "attribute_exists(#body.#nested_error)" \
      --expression-attribute-names '{"#body": "body", "#nested_error": "nested_error"}')
    
    ITEMS=$(echo $SCAN_RESULTS | jq -c '.Items[]')
    
    echo "$ITEMS" | while IFS= read -r ITEM; do
      ITEM_ID=$(echo $ITEM | jq -r '.id.S')
        NESTED_ERROR=$(echo $ITEM | jq -r '.body.M.nested_error.S')
        
        aws dynamodb update-item \
        --table-name $TABLE_NAME \
        --key '{"id": {"S": "'"$ITEM_ID"'"}}' \
        --update-expression "SET #body.#nested = :nested REMOVE #body.#nested_error" \
        --expression-attribute-names '{"#body": "body", "#nested": "nested", "#nested_error": "nested_error"}' \
        --expression-attribute-values '{":nested": {"S": "'"$NESTED_ERROR"'"}}'
    done