I am try to rename and migrate 2TB(10's of millions) of images from a local server onto amazon s3. I wrote PHP to send them using the aws library (running on the same local machine), but I don't have the 6 months it wants to finish. Looking for some ideas on doing this faster.
I have 2 ideas but not sure they would be any better.... 1. Use some mounted s3 solution (s3fs) that will parallel the upload (will it???) 2. Pull the images onto an ec2 instance and send to s3 from there. I could pull them with ssh/FTP or http. Still probably need to parallel them manually??
Any help would be appreciated.
Split your 10M+ list of images into subsets. Upload each subset to S3 in parallel.