amazon-s3google-cloud-platformdata-transfer

How make daily data transfer from AWS s3 to GCP GCS?


I have a client who wants their data to be transferred daily to GCP bucket from their S3 to Google Cloud Storage bucket. Is there any way to create schedule transfer or recurring transfer from s3 to GCS.


Solution

  • you can use the service called Storage Transfer service: https://console.cloud.google.com/transfer

    you can create a pipeline and schedule it as per requirement and run to clone all data from AWS S3 to GCP.

    here main service URL: https://cloud.google.com/storage-transfer-service

    You can use storage transfer service to the transfer bucket data

    it also supports the TSV file.

    You can configure this service to run at a specific time daily.