I am using QuickCheck-2.5.1.1 to do QA. I am testing two pure functions gold :: a -> Float
and f :: a -> Float
, where a
instances Arbitrary.
Here gold
is a reference calculation and f
is a variation I am optimizing.
To date, most of my tests using quickcheck have been using tests like \a -> abs (gold a - f a) < 0.0001
.
However, I would like to gather statistics along with checking the threshold, since knowing the average error and standard deviation are useful in guiding my design.
Is there any way to use QuickCheck to gather statistics like this?
To give a concrete example of the sort of thing I'm looking for, suppose I have the following two functions for approximating square roots:
-- Heron's method
heron :: Float -> Float
heron x = heron' 5 1
where
heron' n est
| n > 0 = heron' (n-1) $ (est + (x/est)) / 2
| otherwise = est
-- Fifth order Maclaurin series expansion
maclaurin :: Float -> Float
maclaurin x = 1 + (1/2) * (x - 1) - (1/8)*(x - 1)^2
+ (1/16)*(x - 1)^3 - (5/128)*(x - 1)^4
+ (7/256)*(x - 1)^5
A test for this might be:
test = quickCheck
$ forAll (choose (1,2))
$ \x -> abs (heron x - maclaurin x) < 0.02
So what I'd like to know, as a side-effect of the test, is the statistics on abs (heron x - maclaurin x)
(such as the mean and standard deviation).
Thanks to the comments from Chris Kuklewicz and Ingo, I came up with the following that collects the statistics I want in my example:
resultToWeightList :: Result -> [(Double,Int)]
resultToWeightList r = [ (read s, n) | (s,n) <- labels r]
weightListMuSigma :: [(Double,Int)] -> (Double,Double)
weightListMuSigma wlst = (mu,sigma)
where
(weightSum,weightSqrSum,entryCount) = foldl addEntry (0,0,0) wlst
addEntry (s,s2,c) (v,w) = (s + (v * w'), s2 + (v**2 * w'), c + w)
where w' = fromIntegral w
entryCount' = fromIntegral entryCount
mu = weightSum / entryCount'
var = weightSqrSum / entryCount' - mu**2
sigma = sqrt var
quietCheckResult :: Testable prop => prop -> IO Result
quietCheckResult p = quickCheckWithResult args p
where args = stdArgs { chatty = False }
test :: IO ()
test = do { r <- quietCheckResult $ forAll (choose (1,2)) test'
; let wlst = resultToWeightList r
; let (mu,sigma) = weightListMuSigma wlst
; putStrLn $ "Average: " ++ show mu
; putStrLn $ "Standard Deviation: " ++ show sigma
}
where
test' x = collect err (err < 0.1)
where err = abs $ heron x - maclaurin x