Friends I need your help, I have to handle a very large file with information (about the file weighs 30 Mb). When processing this information file, java gives me this error:
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2367)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
at java.lang.StringBuffer.append(StringBuffer.java:322)
at java.io.StringWriter.write(StringWriter.java:94)
at java.io.Writer.write(Writer.java:127)
at com.sun.org.apache.xml.internal.serialize.Printer.printText(Printer.java:170)
.....
My code is:
...
public Document generateXML(File file, String type) {
try {
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = dbf.newDocumentBuilder();
Document document = docBuilder.newDocument();
Element rootElement = document.createElement("data");
document.appendChild(rootElement);
Element informacionElement = document.createElement("information");
rootElement.appendChild(informacionElement);
Element entityElement = document.createElement("entity");
entityElement.appendChild(document.createTextNode("ANYTHING"));
informacionElement.appendChild(entityElement);
Element entityCode = document.createElement("code");
entityCode.appendChild(document.createTextNode(0000));
informacionElement.appendChild(entityCode);
Element fileElement = document.createElement("FILENAME");
fileElement.appendChild(document.createTextNode(file.getName()));
informacionElement.appendChild(fileElement);
Element detailElement = document.createElement("detail");
rootElement.appendChild(detailElement);
Path path = Paths.get(file.getAbsolutePath());
Scanner tmpFile = new Scanner(path, StandardCharsets.UTF_8.name());
String[] tmpTags = keyXml.get(type);
int row = 1;
while (tmpFile.hasNextLine()) {
Element rowElement = document.createElement("row");
filaElement.setAttribute("id", Integer.toString(row));
Scanner tmpLine = new Scanner(tmpFile.nextLine());
tmpLine.useDelimiter("\\|");
int index = 0;
while (tmpLine.hasNext()) {
Element element = document.createElement(tmpTags[index]);
String text = tmpLine.next().trim();
element.appendChild(document.createTextNode(text));
filaElement.appendChild(element);
++index;
}
detailElement.appendChild(filaElement);
++fila;
}
return document;
} catch (ParserConfigurationException | IOException ex) {
Logger.getLogger(HandlerXML.class.getName()).log(Level.SEVERE, null, ex);
}
return null;
}
...
This code generates the following XML document:
<data>
<information>
<entity>ANYTHING</entity>
<code>0000</code>
<filename>filename_XXX</filename>
</information>
<detail>
<row id="1">
<tag1>0099</tag1>
<tag2>2014</tag2>
<tag3>05</tag3>
<tag4>0015</tag4>
<tag5>06</tag5>
<tag6>06</tag6>
<tag7>000000000099907</tag7>
<tag8>098026</tag8>
<tag9>00000781.40</tag9>
<tag10>E</tag10>
<tag11>000000006804750</tag11>
<tag12>xxx</tag12>
<tag13>xxxx</tag13>
<tag14>xxx</tag14>
<tag15></tag15>
<tag16></tag16>
<tag17>xx</tag17>
<tag18>xxx</tag18>
<tag19>1014</tag19>
<tag20>05</tag20>
<tag21>11111</tag21>
<tag22>1014</tag22>
<tag23>00000000000000000000</tag23>
<tag24>3</tag24>
<tag25>20101</tag25>
<tag26>00000000</tag26>
<tag27>46671</tag27>
<tag28>1</tag28>
<tag29>N</tag29>
<tag30></tag30>
<tag31></tag31>
<tag32>A</tag32>
<tag33></tag33>
<tag34></tag34>
<tag35></tag35>
<tag36></tag36>
<tag37></tag37>
<tag38></tag38>
</row>
............
</detail>
</data>
word where xml data obtained is:
**0099|2014|05|0015|06|06|00000000003101|066671|000077777.00|E|000000002719415|xxx|xxxxxx |xxxxxxxx | | |xxxxxx | |1014|05|12254758|1014|00000000000000000000|3|20101|00000000|748596|1|X| | |Q| | | | | **
Now imagine that the xml must be generated for 100000 lines of information, and if you are wondering, all the lines must be converted to XML format.
The problem is, when you convert the object into a STRING to DOCUMENT I get the error that I showed earlier, the function I use to transform DOCUMENT to STRING is:
private static String convertDocumentToString(Document doc) {
TransformerFactory tf = TransformerFactory.newInstance();
Transformer transformer;
try {
transformer = tf.newTransformer();
StringWriter writer = new StringWriter();
transformer.transform(new DOMSource(doc), new StreamResult(writer));
return writer.getBuffer().toString();
}
catch (TransformerException e) {}
return "";
}
Now this error is virtual memory and java VM workaround is to increase the memory with -Xmx####m, but I wonder if this is solved by another way, try to handle the data type with: String, StringBuilder and byte. Could anyone solve this question?
Thanks in advance.
"30 MB" isn't really that large.
Suggestion: add this to your Java command line: -XX:MaxPermSize=512m
Also:
Definitely familiarize yourself with the Java profiler, jvisualvm. It ships with the JDK.
Here is a good article for troubleshooting this kind of problem: Anatomy of a PermGen Memory Leak