I am performance testing Milvus 2.4.5 with a distributed setup and a dataset of approximately 5 million items. I'm using HNSW indexing with parameters M=16 and EF construction 100. With a search parameter k=100 and 10K QPS, the Milvus proxy search latency is around 30ms, and the app SDK latency is good.
However, when I increase k to 250, the proxy search latency increases to 40-60ms, but the app SDK search latency spikes to several seconds. Has anyone experienced this issue? What might be the cause, and what parameters should we tune? If the proxy search latency is good, does it mean everything is fine with Milvus, or is there something that needs to be adjusted at the Milvus proxy nodes?
When you increase the topk value from 100 to 250, more id-distance pairs are returned by the proxy node. Each id-distance pair is 12 bytes. At 10K QPS and topk=100, the proxy returns approximately 11.5MB of data per second. Increasing topk to 250 results in about 28.6MB of data per second. This data increase shouldn't typically cause network bandwidth issues unless you have additional output_fields configured, which could exacerbate the problem by returning more data.