Getting an exception com.amazonaws.http.timers.client.SdkInterruptedException on executing below line of code when deployed as a Spark job in AWS EMR.
S3Object response = s3client.getObject(new GetObjectRequest(s3URI.getBucket(), s3URI.getKey()));
The sdk version, code and stack trace are mentioned below. What could be the issue and how to resolve this?
Version Info
Dependency
Code
public static ByteArrayOutputStream getS3Object(String path) {
AmazonS3 s3client = null;
try {
AmazonS3URI s3URI = new AmazonS3URI(path);
s3client = AmazonS3ClientBuilder.defaultClient();
S3Object response = s3client.getObject(new GetObjectRequest(s3URI.getBucket(), s3URI.getKey()));
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = response.getObjectContent().read(buffer)) != -1) {
byteArrayOutputStream.write(buffer, 0, bytesRead);
}
response.close();
return byteArrayOutputStream;
} catch (AbortedException | IOException e) {
e.printStackTrace();
System.err.println("AWS call interrupted retrying " + e);
} finally {
if (s3client != null) {
s3client.shutdown();
}
}
throw new RuntimeException("s3 object is null");
}
Stack trace
com.amazonaws.AbortedException:
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleInterruptedException(AmazonHttpClient.java:880)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:757)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1524)
com.example.S3Utils.getS3Object(S3Utils.java:78)
Caused by: com.amazonaws.http.timers.client.SdkInterruptedException
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:935)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.checkInterrupted(AmazonHttpClient.java:921)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1115)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
... 53 more
You are using an SDK version that is on path to deprecation as mentioned in my comment. Now to get an S3 object using AWS SDK for Java V2, you can use this code. If you do not know how to get up and running with V2, see DEV Guide:
Developer Guide - AWS SDK for Java 2.x
package com.example.s3;
// snippet-start:[s3.java2.getobjectdata.main]
// snippet-start:[s3.java2.getobjectdata.import]
import software.amazon.awssdk.core.ResponseBytes;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
// snippet-end:[s3.java2.getobjectdata.import]
/**
* Before running this Java V2 code example, set up your development
* environment, including your credentials.
*
* For more information, see the following documentation topic:
*
* https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
*/
public class GetObjectData {
public static void main(String[] args) {
final String usage = """
Usage:
<bucketName> <keyName> <path>
Where:
bucketName - The Amazon S3 bucket name.\s
keyName - The key name.\s
path - The path where the file is written to.\s
""";
if (args.length != 3) {
System.out.println(usage);
System.exit(1);
}
String bucketName = args[0];
String keyName = args[1];
String path = args[2];
Region region = Region.US_EAST_1;
S3Client s3 = S3Client.builder()
.region(region)
.build();
getObjectBytes(s3, bucketName, keyName, path);
}
public static void getObjectBytes(S3Client s3, String bucketName, String keyName, String path) {
try {
GetObjectRequest objectRequest = GetObjectRequest
.builder()
.key(keyName)
.bucket(bucketName)
.build();
ResponseBytes<GetObjectResponse> objectBytes = s3.getObjectAsBytes(objectRequest);
byte[] data = objectBytes.asByteArray();
// Write the data to a local file.
File myFile = new File(path);
OutputStream os = new FileOutputStream(myFile);
os.write(data);
System.out.println("Successfully obtained bytes from an S3 object");
os.close();
} catch (IOException ex) {
ex.printStackTrace();
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
}
}
// snippet-end:[s3.java2.getobjectdata.main]
You can find the POM in the AWS Code Example Github here:
https://github.com/awsdocs/aws-doc-sdk-examples/tree/main/javav2/example_code/s3