Is this a good decision to use NoSQL database for a banking system rather than RDBMS?
If yes, What are the recommended NoSQL databases for a banking system?
Nathan Hurst has a really good blog posting on the idea behind NoSQL databases. I'll do my best to paraphrase:
A database is typically chosen based-on properties of Consistency, Availability and Partition Tolerance (CAP Theorem). Of course, the CAP Theorem states that a database can realistically only focus on two of these. NoSQL databases need partition tolerance to scale properly, so they end-up sacrificing either availability or consistency. RDBMSs negotiate this problem by choosing consistency and availability, and utilize other means to keep their data partition tolerant (ex: replication).
You can typically see the effects of this at the transaction-level. In RDBMS-land all transactions should be ACID (Atomic, Consistent, Isolated and Durable). NoSQL databases typically do not have strict ACID requirements. In this way, data that is updated via a transaction may or may not be atomic (transaction is either completed to all update locations or rolled-back), may not be durable if the power fails, and may run under the assumption of "eventual consistency."
Therefore "no", a NoSQL database is definitely not a good idea for a banking solution.
You should also note that "NoSQL" database architectures differ significantly by brand. What I've said here is a generalization about NoSQL databases. It is certainly not all-encomapssing.
EDIT 20241022
A lot has changed in the 12 years since I wrote this answer. I see it's still getting engagement out here, so I felt the need to make a quick addendum.
The argument of banks absolutely need ACID has been thwarted. Today, almost all major banks (in the US) use one or more NoSQL databases. And while some NoSQL databases (like Apache Cassandra®) have found a way to implement ACID transactions, most RDBMSs have adopted a way to support network partition tolerance (either through failover or HA).
So while NoSQL tech of the 2010s may have not been suitable for traditional bank-level transactions, in today's world: