clojure clojure.spec test.check generative-testing

Need help understanding why Clojure spec test/check is failing the return validation when REPL doesn't fail

I've been playing around with Clojure Spec for testing and data generation and am seeing some strange behavior where the function works in unit tests and validation works in REPL but the generative testing with spec.test/check is failing.

I've created a set of specs like so:

(s/def ::significant-string (s/with-gen 
                              (s/and string? #(not (nil? %)))
                              (fn [] (gen/such-that #(not= % "")
                                             (gen/string-alphanumeric)))))

(s/def ::byte-stream 
  (s/with-gen #(instance? java.io.ByteArrayInputStream %)
   (gen/fmap #(string->stream %) (gen/string-alphanumeric))))


(s/fdef string->stream
  :args (s/cat :s ::significant-string)
  :ret ::byte-stream
  :fn #(instance? java.io.ByteArrayInputStream %))

And the fn implementation:

  (defn string->stream
  "Given a string, return a java.io.ByteArrayInputStream"
  ([s] {:pre [(s/valid? ::significant-string s)]
        :post [(s/valid? ::byte-stream %)]}
       (string->stream s "UTF-8"))
  ([s encoding]
   (-> s
       (.getBytes encoding)
       (java.io.ByteArrayInputStream.))))

And in REPL I see what I expect to see from both generation and testing of the specs:

=> (instance? java.io.ByteArrayInputStream (string->stream "test"))
true

=> (s/valid? ::byte-stream (string->stream "0"))
true

=> (s/exercise-fn 'calais-response-processor.rdf-core/string->stream)
([("Y") #object[java.io.ByteArrayInputStream 0x57210dd7 "java.io.ByteArrayInputStream@57210dd7"]] [("d") #object[java.io.ByteArrayInputStream 0x7ec14113 "java.io.ByteArrayInputStream@7ec14113"]] [("5") #object[java.io.ByteArrayInputStream 0x1e85195b "java.io.ByteArrayInputStream@1e85195b"]] [("9c") #object[java.io.ByteArrayInputStream 0x3769ddef "java.io.ByteArrayInputStream@3769ddef"]] [("P0N") #object[java.io.ByteArrayInputStream 0x68793160 "java.io.ByteArrayInputStream@68793160"]] [("7tvN1") #object[java.io.ByteArrayInputStream 0x1cc43ca5 "java.io.ByteArrayInputStream@1cc43ca5"]] [("LjH4U") #object[java.io.ByteArrayInputStream 0x2a3da1a7 "java.io.ByteArrayInputStream@2a3da1a7"]] [("W") #object[java.io.ByteArrayInputStream 0x534287aa "java.io.ByteArrayInputStream@534287aa"]] [("x867VLr") #object[java.io.ByteArrayInputStream 0x72915e93 "java.io.ByteArrayInputStream@72915e93"]] [("moucN3vr") #object[java.io.ByteArrayInputStream 0x4f0d7570 "java.io.ByteArrayInputStream@4f0d7570"]])

But I don't understand why I'm seeing this from the test/check:

(stest/check 'calais-response-processor.rdf-core/string->stream)
    ({:spec #object[clojure.spec.alpha$fspec_impl$reify__2451 0x1acb0d46 "clojure.spec.alpha$fspec_impl$reify__2451@1acb0d46"], :clojure.spec.test.check/ret {:shrunk {:total-nodes-visited 4, :depth 3, :pass? false, :result #error {
     :cause "Specification-based check failed"
     :data {:clojure.spec.alpha/problems [{:path [:fn], :pred (clojure.core/fn [%] (clojure.core/instance? java.io.ByteArrayInputStream %)), :val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :via [], :in []}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$spec_impl$reify__1987 0x16a19b4c "clojure.spec.alpha$spec_impl$reify__1987@16a19b4c"], :clojure.spec.alpha/value {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.test.alpha/args ("0"), :clojure.spec.test.alpha/val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.alpha/failure :check-failed}
     :via
     [{:type clojure.lang.ExceptionInfo
       :message "Specification-based check failed"
       :data {:clojure.spec.alpha/problems [{:path [:fn], :pred (clojure.core/fn [%] (clojure.core/instance? java.io.ByteArrayInputStream %)), :val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :via [], :in []}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$spec_impl$reify__1987 0x16a19b4c "clojure.spec.alpha$spec_impl$reify__1987@16a19b4c"], :clojure.spec.alpha/value {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.test.alpha/args ("0"), :clojure.spec.test.alpha/val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.alpha/failure :check-failed}
       :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
     :trace
     [[clojure.core$ex_info invokeStatic "core.clj" 4739]
      [clojure.core$ex_info invoke "core.clj" 4739]
      ...
      ...(lots more)

It feels like it's related to the generator fn composition although returned object looks "ok" to me right now.

Solution

:fn #(instance? java.io.ByteArrayInputStream %))

The problem is it looks like that :fn spec expects only the function return value, when it's actually being invoked with a map containing the input and return values. Try this version instead:

:fn (fn [{:keys [args ret]}]
      (instance? java.io.ByteArrayInputStream ret))

The :fn spec should be a function that takes a map containing the function's input :args and output :ret value. It's meant to compare the function's output relative to its input.

In this example, the :fn spec seems to be making the same assertion as your :ret spec, and it doesn't look at the :args so you may not want a :fn spec here if there's no meaningful assertion to make between input/output — this would only assert the return value, redundantly.

And in REPL I see what I expect to see from both generation and testing of the specs

The reason you're only seeing the failure with check is because none of those other calls are considering your function's :fn spec e.g. s/exercise-fn doesn't consider the :fn spec.

I made some examples using :fn specs here.