amazon-s3aws-java-sdkawss3transfermanager

Java AWS SDK v1 - S3 API - Not able to upload multiple files parallely using multipart api


I have 5 files of size 200 MB each. I am uploading these files parallel y using Executor service and using TransferManager with multipart threashold = 50 MB and waiting for upload to finish by using blocking method call to upload.waitForCompletion() (which says current thread suspends until upload succeeds or throws error).

Please find below code excerpts:

private static final ExecutorService executor = Executors.newFixedThreadPool(10);

executor.execute(() -> upload("bucketName", new File(fullFilePath)));


public static void upload(final String bucketName, final File filePath) {
        
        AmazonS3 amazonS3 = AmazonS3ClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(getCredentals()))
                .withRegion(Regions.DEFAULT_REGION).build();

        TransferManager tm = TransferManagerBuilder.standard().withS3Client(amazonS3)
                .withMultipartUploadThreshold((long) (50 * 1024 * 1025)).build();
        
        final String fileName = filePath.getName();
        try {
            Upload upload = tm.upload(s3Bucket, fileName, filePath);
            upload.waitForCompletion();
            log.info("Successfully uploaded file = " + fileName);
            
        } catch (Exception e) {
            log.info("Upload failed for file = " + fileName);
            log.error(e);
        }
    }

Main thread does not exits until "Successfully uploaded" is there for all 5 files. Now this program does not throw any error and prints success for all 5 files but when i open the bucket in aws console, nothing is there.

Can anyone suggest what might be happening here or how to further debug it ?


Solution

  • Code is working fine i was looking into the wrong bucket.