I have to migrate about 20 TB data from One S3 bucket to another S3 bucket where both are in separate accounts with separate vendors and I am not the owner in either of them.
I have been provided access to the source and destination buckets with IAM id/secert.
I am looking at either aws CLI or rclone CLI for the transfer.
I am planning to run the CLI commands from a box outside of AWS as I don't have access to either source or destination AWS.
Will the copy or sync move the file directly to destination with AWS or will the files be downloaded from source to my local box and then uploaded to destination ?
I naturally want AWS do the transfer without hitting my box.
Thank you in advance.
To copy objects between S3 buckets, you would need a single set of credentials with Read access on the source bucket and Write access on the destination bucket. This requires co-ordination between the vendors since they both have to work together to provide this access.
If you can achieve this, then you can use aws s3 sync or aws s3 cp --recursive to copy the files directly between buckets.
However, if you cannot use a single set of credentials to access both buckets, then you would need to:
This would best be done from an Amazon EC2 instance in the same region as the S3 buckets, since this will provide the lowest latency and will avoid Data Transfer charges. The EC2 instance doesn't need to be in the same account as the buckets.