javaapache-httpclient-4.xapache-httpasyncclient

Java-Apache BufferedHttpEntity loads whole file when sending the request


I want to make a Multipart request to my server from java using the apache CloseableHttpAsyncClient, this is what i currently have:

File file = new File("pathname");
MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
multipartEntityBuilder.addBinaryBody("file", new FileInputStream(file), ContentType.DEFAULT_BINARY, "file");
multipartEntityBuilder.addTextBody("filename", "File one");
HttpEntity multipartEntity = multipartEntityBuilder.build();
BufferedHttpEntity buffEntity = new BufferedHttpEntity(multipartEntity);

But when uploading a large file it throws a java.lang.OutOfMemoryError and i see it occupy all my machine memory ( Process goes from 100Mb memory to 3000+Mb ).
I have also tried to replace

multipartEntityBuilder.addBinaryBody("file", new FileInputStream(file), ContentType.DEFAULT_BINARY, "file");

with

multipartEntityBuilder.addBinaryBody("file", file);

But this has limited content length and gives the error "Content length is too long".
So how do I upload a large file (50gb+) using apache CloseableHttpAsyncClient?

EDIT

As suggested, i removed the BufferedHttpEntity but now i get the error "Content Length unknown". If i add a header with content length it tells me that content length header is already present.

EDIT 2

If i change the line

multipartEntityBuilder.addBinaryBody("file", new FileInputStream(file), ContentType.DEFAULT_BINARY, "file");

to

multipartEntityBuilder.addPart("file", fileContentBody);

I get the error that Content length is too long


Solution

  • After some trial and error i ended up switching to the CloseableHttpClient with RequestBuilder and HttpUriRequest instead of HttpPost to execute the request in a blocking way on a thread.

    MultipartEntityBuilder multipartEntityBuilder = MultipartEntityBuilder.create();
    multipartEntityBuilder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
    FileBody fileContentBody = new FileBody(file, ContentType.DEFAULT_BINARY, file.getName());
    multipartEntityBuilder.addPart("file", fileContentBody);
    
    multipartEntityBuilder.addTextBody("filename", "File number one");
    HttpEntity multipartEntity = multipartEntityBuilder.build();
    
    
    HttpClientContext context = new HttpClientContext();
    RequestBuilder requestBuilder = RequestBuilder.post("Myurl.com");
    requestBuilder.setEntity(multiPartEntity);
    HttpUriRequest multipartRequest = requestBuilder.build();
    
    HttpResponse response = httpClient.execute(multipartRequest, context);