[SOLVED] Parse large JSON file with JSON Simple (OutOfMemoryError)

Parse large JSON file with JSON Simple (OutOfMemoryError)

I am trying to parse large JSON file with JSON Simple and i am getting out of memory errors. I am on Windows 10 and my laptop has an 8gb RAM. The file is 250mb, i will also need to parse a 2gb file. I also tried with StrinBuilder but then i am getting memory errors on StringBuilder. Here is my code with StringBuilder:

    BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream("myfile.json")));
    String line = null;
    StringBuilder sb= new StringBuilder("");
    while( (line = br.readLine())!= null ){
        sb.append(line);
    }
    JSONParser parser = new JSONParser();
    Object obj=null;
    try {
        obj = parser.parse(sb.toString());  
    }catch (Exception e) {

    }

and here is the code without StringBuilder:

JSONParser parser = new JSONParser();
        Object obj=null;
        try {
            obj = parser.parse(new FileReader("myfile.json"));  
        }catch (Exception e) {

        }

The error

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.json.simple.parser.Yylex.yylex(Unknown Source) at org.json.simple.parser.JSONParser.nextToken(Unknown Source) at org.json.simple.parser.JSONParser.parse(Unknown Source) at org.json.simple.parser.JSONParser.parse(Unknown Source)

Solution

If you are open to use other Json parser then you can try Jackson's Streaming API which can be used to parse huge JSON upto even giga bytes of size.It can be used to process huge files without loading them completely in memory.It allows get the data you want and ignore what you don't want also