I have an application (Springboot api) that runs un kubernetes and i have a single instance with cpu:200m
and memory:8Gi
. My jvm parameters are : _JAVA_OPTIONS: -Xmx7G -Xms7G
My goal is to copy one or more files of size between 30Mb and 1Gb from one s3 bucket to an other with presigned url.
I get java.lang.OutOfMemoryError: Java heap space
error when i call my api more than 2 times in paralelle with 1 gb files.
Here is my code :
public ResponseEntity<String> transferData(Config config) throws IOException {
var inputUrl = config.getInputdUrl();
var outputUrl = config.getOutputdUrl();
var uploadConnection = uploader.getUploadConnection(outputUrl);
try (var inputStream = downloader.getDownloadConnection(inputUrl);
var outputStream = uploadConnection.getOutputStream()) {
IOUtils.copyLarge(inputStream, outputStream);
var resp = uploadConnection.getResponseCode();
return ResponseEntity
.status(HttpStatus.OK)
.body("Transfer done with response:" + resp);
} catch (IOException ioe) {
return ResponseEntity
.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body("An error occurred while transferring data : " + ioe);
}
}
getUploadConnection
return HttpURLConnection
:
var connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
connection.setDoOutput(true);
connection.setRequestMethod(PUT.toString());
return connection;
getDownloadConnection
return an HttpURLConnection
inputStream.
HttpURLConnection connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
connection.setRequestMethod(GET.toString());
connection.setDoOutput(true);
return connection.getInputStream();
Regarding my code and my configuration, I though that the impact on memory would be less important as I stream the file and dont get it completly in memory.
What am i missing ?
EDIT 1:
I added setChunkedStreamingMode
so now my getUploadConnection
return :
var connection = (HttpURLConnection) new URL(url).openConnection();
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setChunkedStreamingMode(-1);
connection.setRequestMethod(PUT.toString());
return connection;
But i get an 411
error from my s3 bucket
With setFixedLengthStreamingMode
set in my outPut connection, I was able to transfer more files without Out of memory error java heap space
issues.
My output Connection :
var connection = (HttpURLConnection) new URL(presignedUrl).openConnection();
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setRequestMethod("PUT");
connection.setFixedLengthStreamingMode(downloadConnection.getContentLengthLong());