i have a question: I want to migrate a buscket structure (folders, subfolders and files) from bucket A to bucket B on a folders created for this migration.
I would like to know: first - if is it possible ? second - can somebody give an idea about how could be the code ? (I'm using pyspark or spark)
I try two ways : using copy_object function using copy function too
and i think that i will have to iterate over all folders one by one and also sub folders too and to finish get file one by one too and create all the structure again on a differente bucket.
Can somebody help me with this.
This is a trivial task which can be accomplished using awswrangler library which you can easily install in GLUE. First get the list of all files that you want to copy from source bucket then use the copy_objects method to replicate them into target bucket
import awswrangler as wr
source_paths = wr.s3.list_objects('s3://source_bucket')
target_paths = wr.s3.copy_objects(paths=source_paths, source_path="s3://source_bucket", target_path="s3://target_bucket")