Recently I've started investigating Couchbase Server as a candidate for the project. A particular scenario I'm looking at right now is to how to make Couchbase acting as a "source of truth" which is why I'm digging into the durability aspect.
So here is a snippet from ACID Properties and Couchbase:
If the durability requirements fail, then Couchbase may still save the document and eventually distribute it across the cluster. All we know is that it didn’t succeed as far as the SDK knows. You can choose to act on this information to introduce more ACID properties into your application.
So imagine next. I insert/update a document and primary node fails until data made it to any replica. Let's say primary is gone for a long time. Now, I don't know at this point whether data was written to disk... So a scary part here is that "Couchbase may still save the document and eventually distribute it across the cluster". Meaning, as far as client can tell, the data didn't make it, so a user would see an error, but then all of a sudden it may appear in the system if primary goes back online.
Am I reading this statement correctly? If I am, what's the best practice to handle it with Couchbase?
So here is what I've found out talking to Couchbase guys:
Scenario #1
One scenario could be that after it has been acknowledged as persisted, but before it’s been replicated, the node fails. In that case, if you do not failover the node, when it comes back online, that item will be replicated.
Scenario #2
One other scenario is that you could have autofailover enabled and after it’s received by the primary but before it’s replicated or persisted, autofailover kicks in and brings a replica to primary. In this case, your application will have seen the failure to achieve the durability requirement requested. If the previous primary does come back online, before it rejoins the cluster it will resync with the state of the current cluster meaning the location where the item is active is now the current state.
So I've asked if "When the former primary gets back online with locally persisted but not replicated items and starts resyncing, would these items get wiped off or something?" -
Yes, and that’s really intentional. You could look at those previously persisted items as an “alternate history” that didn’t play out. When the failure occurred, the cluster picked a new starting place, got everyone to agree, and started the universe moving forward from there. When the old node recovers and tries to join this universe, it has to do so with a shared understanding of that universe, which potentially means dropping data that wasn’t passed along.
Of course, in practice, since replication is memory-to-memory and Disk IO tends to be higher latency (the replication of an item and the persistence of items are both scheduled concurrently), things will tend to be replicated more than persisted, but there is no guarantee. Also, the app (through the SDK) has some ability to influence outcomes too with the Durability Requirements features we were talking about.
The full conversation is here.