OutOfMemory Exception while executing JAVA Mapping

I have to transfer big files (500 MB+...can also be of 1GB in size). These files have to be base64 encoded and the encoded string has to be put in a XML file. While my below code works good for smaller files (30 - 50 MB) it fails for files great than 100 MB.

I am using base64 encoder from SUN (sun.misc.BASE64Encoder).

public void execute(InputStream inputstream, OutputStream outputstream) throws StreamTransformationException{
        try
        {
            String sourceFileName = "test_file";
            String ReceiverStr = "";
            //2. Convert input data in Base64Encoded string
            BASE64Encoder encoder = new BASE64Encoder();
            byte input[] = new byte[inputstream.available()];

            inputstream.read(input);
            String base64Encoded = encoder.encode(input);
            //3. Build the SOAP request format
            String serverUrl = "http://website/url";
           
            String soapEnvelope = "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:soap=\"http://schemas.microsoft.com/sharepoint/soap/\">";
            String soapHeader = "<soapenv:Header/><soapenv:Body><soap:CopyIntoItems><soap:SourceUrl>C:\\Users\\Desktop\\test_file.txt</soap:SourceUrl><soap:DestinationUrls><soap:string>" + serverUrl + "</soap:string></soap:DestinationUrls><soap:Fields><soap:FieldInformation " + "Type=" + "\"Text\"" + " DisplayName=\"" + sourceFileName + "\"" + " InternalName=\"" + sourceFileName + "\"" + " Id=\"deff4b5c-b727-414c-893d-c56a8e12455f\"" + " Value=\"" + sourceFileName + "\"/></soap:Fields>";
            String soapStream = "<soap:Stream>" + base64Encoded + "</soap:Stream>";
            ReceiverStr = soapEnvelope + soapHeader + soapStream + "</soap:CopyIntoItems></soapenv:Body></soapenv:Envelope>";
            //4. Write the SOAP request to receiver channel
            outputstream.write(ReceiverStr.getBytes());
        }
        catch(Exception e) {
            throw new StreamTransformationException(e.toString());  
        }
    }

When I try to see the message at run-time, then the entire message is not displayed and it is truncated in-between in the base64Encoded string. Below is the error that is seen in my system on executing the JAVA code.

Please note that my server settings can otherwise easily transfer 1GB+ files without any JAVA Heap size error or file truncation.

Can you please let me know how can I process big files using above logic?

Thanks,

Abhishek.

Taking a look at your code you store the data three times:

The first time is in byte array from the input stream. The second time is in the encoded String and finally you write it to the output stream.

What you can do to optimize this a bit is to split the reading from the input stream/encoding and the writing of the encoded string to the output stream. That way you can let go of the input byte array once encoded and the memory can be freed. An even better solution would be when the encoder directly writes to the Output stream or directly write to an encoding output stream.

However you will still have to consider what the maximum file size should be that can be processed and adjust the heap setting accordingly: How to deal with "java.lang.OutOfMemoryError: Java heap space" error (64MB heap size)