I have this jsfiddle where I'm trying to measure performance for some operation of my interest, and I'm constantly getting different accuracy for individual tests. For example here is my last run:
using extra assign x 662 ops/sec ±9.20% (57 runs sampled)
using no extra assign x 694 ops/sec ±4.31% (47 runs sampled)
Fastest run(s) is(are): "using no extra assign,using extra assign"
It's hard to measure results that have different accuracy, which in some cases can be as much as ±30.0%, and so I would like to be able to set the acceptable boundary for accuracy.
I was trying to tweak options but they seem like not affecting anything or I'm doing something wrong. What am I doing wrong?
Thank you in advance!
Adding the minSamples option to each benchmark and setting it to a higher level will force benchmark.js to take more samples for each benchmark. Having the larger sample set should reduce the margin of error (MOE). Play around with the sample size to determine a desired balance between accuracy and test duration.
This is an example using the code you provided in the jsfiddle:
suite
.add('using extra assign', test1, { minSamples: 200 })
.add('using no extra assign', test2, { minSamples: 200 })
From these changes, the following results are produced:
using extra assign x 642 ops/sec ±0.74% (240 runs sampled)
using no extra assign x 637 ops/sec ±0.44% (258 runs sampled)
Fastest run(s) is(are): "using extra assign"