androidamazon-web-serviceskotlinamazon-dynamodbaws-amplify-sdk-android

How to upload bulk data to AWS DynamoDB with BatchWriteItem or Batch put custom resolver In Android Kotlin


I have a Android project in Kotlin which uses AWS amplify services like Cognito, Datastore, Appsync etc.., I am a beginner to AWS concepts and the problem I am facing from a month is with the bulk data sync to dynamoDB, here i want to sync around 2000 records to dynamoDB with minimum time delay.

What I tried Initially is to add the data to Datastore and let the appsync take care of data sync to dynamodb like bellow, But it took around 20mins to sync 2000 records with dynamoDB.

   for (i in 1..2000) {
        val item = RandomNumbers.builder()
            .number(i)
            .build()
        Amplify.DataStore.save(item,
            {
                Log.d("TAG", "number saved:" + it.item().number)
                if (it.item().number == 1999) {
                    Log.e("TAG", "Ended: " + System.currentTimeMillis())
                }
            },
            {
                Log.d("TAG", "Failed to save number")
            })
    }

The second approach i used was Amplify.API.mutate which took around 200 seconds to sync 2000 records to dynamodB

    for (i in 1..2000) {
        val item = RandomNumbers.builder()
            .number(i)
            .build()
        Amplify.API.mutate(
            ModelMutation.create(item),
            { Log.i("MyAmplifyApp", "Todo with id: ${it.data}") },
            { Log.e("MyAmplifyApp", "Create failed", it) }
        )
    }

The problem with all the above methods is that, these methods sync the records one by one Sequentially to the dynamoDb which result in huge time delay

I found that AWS has a BatchWriteItem and Batch put custom resolver both of this approach seems to be what i was exactly looking for, But the problem is, I did not find any documentation or any sample code to use Batch put custom resolver or BatchWriteItem to implement the Batch Upload operation in Android. ( I found the sample code for javascript & java server side implementation in aws example code git repo aws-doc-sdk-examples )

Can anyone please guide me on how i can implement this Batch Upload operation in Android ?


Solution

  • I was able to implement Batch upload to DynamoDb tables in Android by using BatchWriteItemRequest

        val dynamoDbClient = DynamoDbClient {
            region = "us-east-1"
            credentialsProvider = <CREDENTIAL PROVIDER>
        }
    
        //creating 2000 items to upload  
        val numbersList =  mutableListOf<RandomNumbers>()
        for (i in 1..2000) {
            val items = RandomNumbers.builder()
                .number(i)
                .id(UUID.randomUUID().toString())
                .build()
            numbersList.add(items)
        }
    
    
        val batchSize = 25 //For BatchWriteItemRequest the limit for items in a batch is 25 
        val batches = numbersList.chunked(batchSize)
    
        // Iterate through each batch and process them
        batches.forEachIndexed { _, batch ->
            // Construct batch write request for each batch and execute
            val batchWriteItemRequest = BatchWriteItemRequest {
                requestItems = mapOf(
                    "RandomNumbers-f4v36ohral7pym-dev" to batch.map { value ->
                        WriteRequest {
                            putRequest =  PutRequest {
                                item = mapOf(
                                    "number" to AttributeValue.N(value.number.toString()),
                                    "id" to AttributeValue.S(value.id.toString()),
                               
                                )
                            }
                        }
                    }
                )
            }
            val batchWriteResponse = dynamoDbClient.batchWriteItem(batchWriteItemRequest)
            Log.e("TAG", "response: "+batchWriteResponse.toString() )
            
        }
    

    The above code will create 2000 records and insert them to AWS DynamoDb Table that is RandomNumbers-f4v36ohral7pym-dev in a Batch of 25 each (the batch upload item limit set by aws)

    I have used bellow dependencie

    implementation("aws.sdk.kotlin:dynamodb:1.0.65")