marklogicmarklogic-9ml-gradlemarklogic-corb

Setting heap size for CorbTask

In my build.gradle, I have a task of type CorbTask, where I am looking to execute some process over a large number of URIs. In this case, I am processing a little less than 800,000 URIs. 

task updateSharePointMetadata(type: com.marklogic.gradle.task.CorbTask) {
  classpath = configurations.corb
  xccUrl = "xcc://${mlUsername}:${mlPassword}@${mlHost}:${mlStagingPort}"
  urisModule = "corb/selectSharepointMetadata.xqy|ADHOC"
  processModule = "corb/updateBinarySizeInSharePointMetadata.xqy|ADHOC"
  threadCount = 8
  batchSize = 100
  batchUriDelim=','
}

When I run this task... during the process of queuing up the URIs, I get a OutOfMemoryError, Out of Heap space. When the task is run... or at least I did earlier, now at least it runs.. but I still see the Free Memory warning as it queues up the uris.

WARNING: free memory: 35 MiB of 394

In the gradle properties I already have the org.gradle.jvmargs parameter setup to pass in the -Xmx setting, but it does not seem to make any difference. I assume this is because the CorbTask extends JavaExec, which is spawned off. So I looked at maybe setting the jvmArgs property on the CorbTask, but that is not settable.

Does anyone have any ideas?

Solution

  • The DISK-QUEUE option was added for scenarios such as this.

    Rather than trying to bump up your memory settings and hope that a large set of URIs will fit, you can enable the DISK-QUEUE option.