I am looking for a java example that shows how to avoid exceptions with Membase when one of its nodes goes down.
I have a small cluster of two nodes with one 'default' bucket. It is replicated on both servers. I wrote little java test app for stress loading. I use spymemcache 2.7. When I run it - both servers get busy. When I shutdown one membase instance my java app crushes.
Here is exception log:
2011-06-15 17:32:33.405 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/192.168.1.9:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2011-06-15 17:32:33.407 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/192.168.1.10:11211, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2011-06-15 17:32:33.412 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@63238bd2
2011-06-15 17:32:33.413 INFO net.spy.memcached.MemcachedConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@37bd2664
2011-06-15 18:20:21.896 INFO net.spy.memcached.MemcachedConnection: Reconnecting due to exception on {QA sa=/192.168.1.9:11211, #Rops=2, #Wops=0, #iq=0, topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@5f4275d4, topWop=null, toWrite=0, interested=1}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
at sun.nio.ch.IOUtil.read(IOUtil.java:210)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:487)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:427)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:280)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:2063)
2011-06-15 18:20:21.897 WARN net.spy.memcached.MemcachedConnection: Closing, and reopening {QA sa=/192.168.1.9:11211, #Rops=2, #Wops=0, #iq=0, topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@5f4275d4, topWop=null, toWrite=0, interested=1}, attempt 0.
2011-06-15 18:20:21.898 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: net.spy.memcached.protocol.binary.StoreOperationImpl@5f4275d4
2011-06-15 18:20:21.899 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: net.spy.memcached.protocol.binary.GetOperationImpl@802b249
Exception in thread "main" java.lang.RuntimeException: Exception waiting for value
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1146)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1163)
I made node 192.168.1.9 go down, but client didn't understood that and crashed.
Any ideas how to properly handle it?
So, we didn't find answer for the question and we don't use this software anymore. Now we use Erlang with absolutely illuminates the need in such kind of in-memory data storage.