amazon-web-servicesamazon-s3awss3transfermanager

S3 transfer manager takes huge time to upload


I am uploading my application objects to s3 via transfer manager. It works fine except rare cases where it takes as huge as 15 minutes for an 300 KB object size. I am just exploring to introduce retry logic so that I can say after 30 seconds abort current upload and resubmit it. Below is my code

private static UploadResult put(
            final AmazonS3 amazonS3,
            final Supplier<PutObjectRequest> requestSupplier) throws Exception {
        TransferManager tx = TransferManagerBuilder
                .standard()
                .withS3Client(amazonS3)
                .build();
        try {
            Upload myUpload = tx.upload(requestSupplier.get());

            return myUpload.waitForUploadResult();
        } finally {
            tx.shutdownNow(false);
        }
    }

Can we have something which can allow me to timeout this upload request after say 30 seconds so that I can retry again. waitForUploadResult method is a blocking call so cant do anything about it but wait for it to complete.

Thanks in advance!


Solution

  • I couldn't find a way to timeout the upload request so I submitted it in an executor so waiting part runs in separate thread. Below is my code

    private static UploadResult put(
                final AmazonS3 amazonS3,
                final Supplier<PutObjectRequest> requestSupplier) throws Exception {
            TransferManager tx = TransferManagerBuilder
                    .standard()
                    .withS3Client(amazonS3)
                    .build();
            ExecutorService ex = Executors.newSingleThreadExecutor();
            Upload myUpload = tx.upload(requestSupplier.get());
            try {
                Future<UploadResult> f = ex.submit(() -> myUpload.waitForUploadResult());
                return f.get(2, TimeUnit.MINUTES);
            } catch (TimeoutException te) {
                myUpload.abort();
                throw new RuntimeException("S3 save operation timeout after 2 minutes. Aborted.", te);
            } finally {
                ex.shutdownNow();
                tx.shutdownNow(false);
            }
        }