I need to backup my aws s3 bucket with lots of files (800k files, 400GB) to google cloud storage. Google Storage Transfer Service seems like an easy solution but i have some doubts about cost-efficiency.
First time it will download a whole s3 bucket:
What about the second time and later?
It will cost the same as first time or it will download only changed/new files?
Per the documentation at Creating and Managing Data Transfers with the Console: "By default, Storage Transfer Service only overwrites an object when the source version is different from the sink version."
Therefore, the cost of the second time will be the cost of the list operations on the S3 and GCS buckets (needed to compare to see what changed), plus GETs and PUTs for whatever changed. I believe objects are typically listed 1,000 at a time (according to the maxmimum page size for object listings in each service). Therefore, you could expect the cost in your scenario would be approximately 800 list operations in S3 and 800 list operations in GCS for each subsequent transfer if no objects changed.