javajsonspringspring-bootlarge-file-upload

Upload a json containing base 64 encoded file(size 400Mb) to spring REST interface using curl command

I am trying to upload a json containing large base64 encoded file(size: 400Mb) to a Spring REST interface using curl. But I receive a OutOfMemory error. I understand that the json containing the file is exceeding the JVM's heap size. What approach would be good to process such a large file? Any sample processing code would be much appreciated.

Shell commands:

jsonstring='{"uuid":"111","type":"REPORT","userdata":"test defined 
data","time":3000,"wasted":120,"status":"PASS","message":"demo 
message","report":"'"$(base64 file.zip)"'"}'
curl -s --insecure -H "Content-Type: application/json;" --data "@-" 
https://localhost:443/api/v1/upload <<<"$jsonstring"

Spring Rest Interface :

@RequestMapping(value = "/api/v1/upload", method = RequestMethod.POST)
public String uploadFile(@RequestBody final UploadedFile uploadedFile, final 
HttpServletRequest request,
final HttpServletResponse response) {
byte[] decoded = 
DatatypeConverter.parseBase64Binary(uploadedFile.getReport());

Error :

Jul 17 10:03:38  bash[29824]: 2018-07-17 10:03:38.686 ERROR 29829 --- [.0-31443-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
Jul 17 10:03:38 chld9004852 bash[29824]: java.lang.OutOfMemoryError: Java heap space
Jul 17 10:03:38 chld9004852 bash[29824]: at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68) ~[na:1.8.0_152]
Jul 17 10:03:38 chld9004852 bash[29824]: at java.lang.StringBuilder.<init>(StringBuilder.java:101) ~[na:1.8.0_152]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:394) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2408) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:269) ~[jackson-core-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:35) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:127) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Jul 17 10:03:38 chld9004852 bash[29824]: at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:369) ~[jackson-databind-2.9.6.jar!/:2.9.6]
Solution

  • Your main problem is Spring using the Jackson mapper to load the whole thing into memory. You can only avoid this by using the lower-level Streaming API JSON parser -- think of XML SAX or StAX vs DOM.

    The idea is to make your controller expect an InputStream instead of a mapped object and parse it by hand, one token at a time . This way you're able to only create the objects or data structures required for processing that request, rather than loading the entire thing plus quite a bit of overhead.

    The other option is, of course, to just increase the available heap of the JVM, and wait for the next huge file to crash it. ;)