unit-testingscalatestscalacheckproperty-based-testing

Does property based testing make you duplicate code?

I'm trying to replace some old unit tests with property based testing (PBT), concreteley with scala and scalatest - scalacheck but I think the problem is more general. The simplified situation is , if I have a method I want to test: 

 def upcaseReverse(s:String) = s.toUpperCase.reverse

Normally, I would have written unit tests like: 

assertEquals("GNIRTS", upcaseReverse("string"))
assertEquals("", upcaseReverse(""))
// ... corner cases I could think of

So, for each test, I write the output I expect, no problem. Now, with PBT, it'd be like : 

property("strings are reversed and upper-cased") {
 forAll { (s: String) =>
   assert ( upcaseReverse(s) == ???) //this is the problem right here!
 }
}

As I try to write a test that will be true for all String inputs, I find my self having to write the logic of the method again in the tests. In this case the test would look like :

   assert ( upcaseReverse(s) == s.toUpperCase.reverse)

That is, I had to write the implementation in the test to make sure the output is correct. Is there a way out of this? Am I misunderstanding PBT, and should I be testing other properties instead, like :

That is also plausible but sounds like much contrived and less clear. Can anybody with more experience in PBT shed some light here?

EDIT : following @Eric's sources I got to this post, and there's exactly an example of what I mean (at Applying the categories one more time): to test the method times in (F#): 

type Dollar(amount:int) =
member val Amount  = amount 
member this.Add add = 
    Dollar (amount + add)
member this.Times multiplier  = 
    Dollar (amount * multiplier)
static member Create amount  = 
    Dollar amount

the author ends up writing a test that goes like: 

let ``create then times should be same as times then create`` start multiplier = 
let d0 = Dollar.Create start
let d1 = d0.Times(multiplier)
let d2 = Dollar.Create (start * multiplier)      // This ones duplicates the code of Times!
d1 = d2

So, in order to test that a method, the code of the method is duplicated in the test. In this case something as trivial as multiplying, but I think it extrapolates to more complex cases.

Solution

  • This presentation gives some clues about the kind of properties you can write for your code without duplicating it.

    In general it is useful to think about what happens when you compose the method you want to test with other methods on that class:

    For example:

    Then think about what would break if the implementation was broken. Would the property fail if:

    1. size was not preserved?
    2. not all characters were uppercased?
    3. the string was not properly reversed?

    1. is actually implied by 3. and I think that the property above would break for 3. However it would not break for 2 (if there was no uppercasing at all for example). Can we enhance it? What about:

    I think this one is ok but don't believe me and run the tests!

    Anyway I hope you get the idea:

    1. compose with other methods
    2. see if there are equalities which seem to hold (things like "round-tripping" or "idempotency" or "model-checking" in the presentation)
    3. check if your property will break when the code is wrong

    Note that 1. and 2. are implemented by a library named QuickSpec and 3. is "mutation testing".

    Addendum

    About your Edit: the Times operation is just a wrapper around * so there's not much to test. However in a more complex case you might want to check that the operation:

    If any of these properties fails, this would be a big surprise. If you encode those properties as generic properties for any binary relation T x T -> T you should be able to reuse them very easily in all sorts of contexts (see the Scalaz Monoid "laws").

    Coming back to your upperCaseReverse example I would actually write 2 separate properties:

     "upperCaseReverse must uppercase the string" >> forAll { s: String =>
    upperCaseReverse(s).forall(_.isUpper)
 }

 "upperCaseReverse reverses the string regardless of case" >> forAll { s: String =>
    upperCaseReverse(s).toLowerCase === s.reverse.toLowerCase
 }

    This doesn't duplicate the code and states 2 different things which can break if your code is wrong.

    In conclusion, I had the same question as you before and felt pretty frustrated about it but after a while I found more and more cases where I was not duplicating my code in properties, especially when I starting thinking about