[SOLVED] AWS DMS s3 target folder appear and disapear

AWS DMS s3 target folder appear and disapear

I have a DMS replication task on-going from RDS Aurora (MySQL) to S3.

My S3 endpoint settings are

{
    "CsvRowDelimiter": "\\n",
    "CsvDelimiter": ",",
    "BucketFolder": "dms",
    "BucketName": "mybucketname",
    "CompressionType": "NONE",
    "EncryptionMode": "SSE_KMS",
    "ServerSideEncryptionKmsKeyId": "arn:aws:kms:xxxxxxxxxxxxxxxx",
    "EnableStatistics": true,
    "IncludeOpForFullLoad": true,
    "CdcInsertsOnly": false,
    "TimestampColumnName": "TIMESTAMP",
    "DatePartitionEnabled": true,
    "DatePartitionSequence": "yyyymmdd",
    "DatePartitionDelimiter": "slash",
    "AddColumnName": true,
    "Rfc4180": true
}

I have some folders in my s3 who appears and disappear. On this particular folders, I don't have the LOAD00000001.csv and only my folder structure of the day (let's say now /2023/01/16/ and file within it.

Is this a normal behavior? If yes, how can I fix it? I'm waiting to have nothing deleted at all, even in case of schema/DDL changes.

Thanks

Solution

I had a similar issue and it has to do with ChangeProcessingDdlHandlingPolicy in task settings (JSON). The default is true, meaning tables will be removed from S3 when a table is loaded in truncate and insert fashion. Check out the below link.

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.DDLHandling.html