Currently running a big data job that is doing lots of small inserts into an accumulo table, however after running for about an hour will always get an OOM exception
2018-10-09 12:19:17,345 [rpc.CustomNonBlockingServer$CustomFrameBuffer] WARN : Got an IOException in internalRead!
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:142)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:539)
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:203)
at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154)
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
# Executing /bin/sh -c "kill -9 71952"...
Killed
Is there a way to increase the memory when you start off the proxy server either through the command line or the config file? their documentation is spare and doesnt off much insight.
./bin/accumulo proxy -p proxy/proxy.properties #option to increase memory
The bin/accumulo command you're executing is a bash shell script that sources conf/accumulo-env.sh where you'd normally set some Java command-line options.
The provided launch scripts are a bit confusing in versions prior to Accumulo 2.x, but is anticipated to be much simpler and more direct/intuitive in 2.0.0 and later. In 1.9.x (and maybe some earlier versions), you can add appropriate Java flags to increase memory to the ACCUMULO_OTHER_OPTS environment in the conf/accumulo-env.sh file.