[SOLVED] Initialize benchmark in criterion and exclude initialization time from results

Initialize benchmark in criterion and exclude initialization time from results

I need to benchmark some code inside IO, and criterion supports that pretty well. But I want to perform few initialization steps (different for each benchmark). The naive approach:

main = defaultMain
  [ bench "the first" $ do
      initTheFirst
      theFirst
      cleanUpTheFirst
  , bench "the second" $ do
      initTheSecond
      theSecond
      cleanUpTheSecond
  ]

But it performs initialization and cleanup for every benchmark run (100 times by default) and includes initialization time to final results. Is it possible to exclude initialization time?

ADDED: The code uses global state (mongodb actually), so I can't prepare two initial states simultaneously.

Solution

Three real options I can see here. Either initialize and cleanup before and after:

main = do
  initTheFirst
  initTheSecond

  defaultMain
    [ bench "the first" theFirst
    , bench "the second" theSecond
    ]

  cleanUpTheFirst
  cleanUpTheSecond

Or, if that isn't possible, also benchmark your cleanup and initialization processes and modify your benchmark times accordingly.

Alternatively, you could forego using the provided defaultMain and instead roll your own using the lower-level functions provided by Criterion.