redisdistributed-system

Why are hotspots on my redis cluster bad?


I have a redis cluster and I am planning to add keys which I know will have a much heavier read/update frequency than other keys. I assume this might cause hotspots on my cluster. Why is this bad and how can I avoid it ?


Solution

  • Whether or not items will cause a hot spot on a particular node or nodes depends on a bunch of factors. As already mentioned, hotspotting on a single key isn't necessarily a problem if the overall cluster traffic is still relatively even and the node that key is on isn't taxed. If each of your cluster nodes are handling 1000 commands/sec and on one of those nodes all of the commands are one related to one key, it's not really going to matter since all of the commands are processed serially on a single thread, it's all the same.

    However, if you have 5 nodes, all of which are handling 1000 commands/sec, but you add a new key to one node which makes that single node incur another 3000 commands/sec, one of your 5 nodes is now handling 50% of the processing. This means that it's going to take longer for that node handle all of its normal 1000 commands/sec, and 1/5 of your traffic is now arbitrarily much slower.

    Part of the general idea of distributing/sharding a database is not only to increase storage capacity but to balance work as well. Unbalancing that work will end up unbalancing or screwing up your application performance. It'll either cause 1/Nth of your request load to be arbitrarily slower due to accessing your bogged down node, or it'll increase processing time across the board if your requests potentially access multiple nodes. Evenly spreading load gives an application better capacity to handle higher load without adversely effecting performance.


    But there's also the practical consideration of whether the access to your new keys are proportionally large to your ongoing traffic. If your cluster is handling 1000+ commands/sec/node and a single key will add 10 req/sec to a single particular node, you'll probably be fine just fine either way.