Error in Integration Studio when calling endpoint that returns a lot of data

I am trying to call a rest api that returns a maximum of 1000 data in wso2 integration studio and list its results in the script mediator. It does not work when the limit parameter is set to a value close to 1000. I can fetch 500 pieces of data. But I can't fetch more. How can I get this to work when there is too much data?

The error I got:

The script engine returned an Exception executing the external js script : null function mediate java.lang.IllegalArgumentException: out of range index

Here is my sample work:

<resource methods="GET" url-mapping="/GetTotalBuildingCountAndArea">
        <inSequence>
            <call>
                <endpoint>
                    <http method="get" uri-template="http://localhost/entities?type=Building&amp;limit=800">
                        <suspendOnFailure>
                            <initialDuration>-1</initialDuration>
                            <progressionFactor>-1</progressionFactor>
                            <maximumDuration>0</maximumDuration>
                        </suspendOnFailure>
                        <markForSuspension>
                            <retriesBeforeSuspension>0</retriesBeforeSuspension>
                        </markForSuspension>
                    </http>
                </endpoint>
            </call>
            <script language="js"><![CDATA[
            
            var buildings = mc.getPayloadJSON();
            mc.setPayloadJSON(JSON.stringify(buildings));
            
            ]]></script>
            <property name="messageType" scope="axis2" type="STRING" 
               value="application/json"/>
            <jsontransform description="Json Convert">
                <property name="synapse.commons.json.output.autoPrimitive" value="true"/>
                <property name="synapse.commons.enableXmlNullForEmptyElement" 
             value="false"/>
            </jsontransform>
            <respond/>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </resource>

Solution

This is a limitation of Script Mediator. The documentation has this to say.

The Micro Integrator uses Rhino engine to execute JavaScripts. Rhino engine converts the script to a method inside a Java class. Therefore, when processing large JSON data volumes, the code length must be less than 65536 characters, since the Script mediator converts the payload into a Java object. However, you can use the following alternative options to process large JSON data volumes.

Achieve the same functionality via a Class mediator .

If the original message consists of repetitive sections, you can use the Iterate mediator to generate a relatively small payload using those repetitive sections. This will then allow you to use the Script mediator.

The Script Mediator supports using Nashorn to execute JavaScripts, in addition to its default Rhino engine.

It's best to handle this in such a way you altogether avoid the issue. But you can try the third option mentioned above as well. In the Script mediator try changing the language to nashornJS.

<script language="nashornJS">
.
.
.