ExpressJS app which receives around 70 requests per second - slow Cassandra performance

This is not a question related to a code, but more to a server performance and things that I should check. So I have a ExpressJS server which is connected to a cassandra db (1 seed node and 2 nodes on 1 cluster, so in total 3 nodes). The API is running on the same server as the cassandra db seed node. I have in total 3 servers in local network.

So the structure looks like this -

server 1 running API and seed cassandra node. server 2 running cassandra node. server 3 running cassandra node.

Each server has 8GM of ram and 2.5Ghz CPU.

By default there are around 70 requests coming in each second which does the following -

1) Calls a function which reads the data from a table from the cassandra (using materialized view). 2) Reads another table from cassandra db (using materialized view). 3) Posts data to the third table in cassandra.

The 2nd function that is called is quite similar, it does 1 read using materialized view and 1 post.

The proportional difference between the function called each second is around 30 times function 1 is called (which does 2 reads and 1 post), and around 40 times function 2 is called (which does 1 read and 1 post).

Everything would be great, but the latency of the requests are jumping from time to time, sometimes it takes around 10ms, but every 5 - 10 seconds it goes up to 3-30 seconds. Also cassandra seems to be unstable - during the period when there are 3-30 second request times, the cassandra seem to time out on some of the requests.

What would be the first thing that I should check? Do I need additional nodes and how can I figure out if I have enough nodes for the amount of data being sent in to cassandra db? Should I separate API from cassandra nodes - thus hold API server on a separate server, e.g., server 4?

Solution

Materialized views are great for reads operations, but they come with the expense of writes; you'll need to account for some overhead required to execute its magic:

materialized view will require additional resources to track updates on its sources; this will get worse when you interact with multiple materialized views, as in the first scenario that you propose.
If the data of the post is written in the same source of the materialized view, this will depend on the complexity of the primary key used in the table, as explained here.

The first option that I would explore is to denormalize and create a separate table for the first function, so you will do one read, instead of two.

There is a lot of speculation in my answer, as there are a lot of unknowns on the structure and your table schema; you may get a better insight if you enable tracing, on our case we have gotten good results with openzipkin as explained by TLP