Sustained high latencies in Cloud Spanner

I am using Google Cloud Spanner with my Application which is latency sensitive. I see sustained high median latencies for all/most of my requests. My CPU utilization is less than 40%? How do I fix it?

Common scenarios for high latencies and low CPU are:

1. Lock Contention : To ensure the consistency of multiple concurrent transactions, Spanner uses locks to control access to the data. Lock contention occurs when concurrent transaction access the same data, leading to high transaction latencies for reads and writes. Debugging of lock contention can be done using the Lock and Transaction Insights.
2. Hotspotting : Check if there are any hotspots in the database, which can be verified using the Key Visualiser. If there are indeed any hotspots, refer to the Schema Design Best Practices to avoid them.
3. Large Transactions : Large transactions (transaction that change many tables, many rows in the same table that are not adjacent or tables with a lot of indexes) can lead to higher latencies. Data needs to be read and written from the participants in the transaction so a higher participant count will mean more network overhead for coordination and additionally more processing will be done in the same transaction. The participants count for transactions can be found in the Transaction Details page of Transaction Insights.