distributed-transactionscap-theorem

Using transaction history instead of balance in distributed transactional systems


Preamble

CAP theorem states that we cannot have strictly consistent and available distributed system. For transactional systems (e.g. some payment system) consistency is usually prefered over availability as we cannot tolerate cases where money are "created" or "destroyed".

Example

Both A and B want to transfer $10 to C. The initial balance of C is $15. If both A and B simultaneously read current balance of C and add $10 then C will be $25, but it should be $35 (15+10+10)

Question

Is it a good practice to use transaction history and compute current balance instead of storing balance? What are pros and cons of such approach?

I was reading\watching several sources about consistency and distributed systems but I haven't found anything about this.

Thoughts

Pros:

  1. It's impossible to "Create"/"Destroy" money.
  2. No need to sync with other nodes in order to commit a transaction.

Cons:

  1. Overdraft is possible.
  2. Probably more computation in order to display balance. Although it can be mitigated by caching balance for some point in the past.

So it seems to me that such system will be kind of middle point between consistency and availability...


Solution

  • What you are describing is more formally known as an operational CRDT.

    Yes, it can work, in the way that mutation operations are performed. There are, generally, several downsides to this approach.

    The first one is that operations are not idempotent (because you are applying a difference/delta to a balance in your example, rather than setting a balance).

    Second, as the system has greater uptime, the delta of changes will accumulate. If at any point a new participant wants to become aware of the balance which you are referring to, they need to read ALL history and re-calculate the current balance. This could be a problem if you are dealing with a large number of changes.

    In order to avoid this, some form of snapshotting is typically done. However, this can be difficult to build because of the third point.

    Third, your system will not guarantee at any point of time that your participants have a single coherent view of the balance.

    Now, depending on what you want to do, I can think of some solutions to address these concerns.

    You could, for example, supply an identifier (UUID) to each of your updates. That would give you some level of confidence that your updates can be applied idempotently. The identifiers can also then be used as a way of snapshoting (all balance until id X is 1000 for example). A new participant in your network will then just have to read the operational history up to the last snapshot.

    Snapshot creation is going to be a bit of a problematic process, however, because of your system: ALL participants will have to agree that the balance up to X is some value. At this point, you may discover that some participants miss pieces of history, and not just the latest updates, and so they must be re-synced.