I have a cluster of 10 memcaches, using consistent hashing. When the key passed to memcached_get() is searched on the unavailable server I get just MEMCACHED_SERVER_MARKED_DEAD response (return value).
I would expect the key should be redistributed to the next available server in this case and I should get NOTFOUND from the next memcached_get() call. However I'm still getting MEMCACHED_SERVER_MARKED_DEAD and so I'm unable to set a new value.
I discovered I can call memcached_behavior_set(..., MEMCACHED_BEHAVIOR_DISTRIBUTION). This causes hash redistribution and it works as I wish then. However, I do not think it is a good approach. Is it?
I found the answer myself.
https://bugs.launchpad.net/libmemcached/+bug/777672
Applying the patch solved all my problems. Note, I wonder it has beed broken since 0.39 and nobody has cared.