I've been playing around with Clojure Spec for testing and data generation and am seeing some strange behavior where the function works in unit tests and validation works in REPL but the generative testing with spec.test/check is failing.
I've created a set of specs like so:
(s/def ::significant-string (s/with-gen
(s/and string? #(not (nil? %)))
(fn [] (gen/such-that #(not= % "")
(gen/string-alphanumeric)))))
(s/def ::byte-stream
(s/with-gen #(instance? java.io.ByteArrayInputStream %)
(gen/fmap #(string->stream %) (gen/string-alphanumeric))))
(s/fdef string->stream
:args (s/cat :s ::significant-string)
:ret ::byte-stream
:fn #(instance? java.io.ByteArrayInputStream %))
And the fn implementation:
(defn string->stream
"Given a string, return a java.io.ByteArrayInputStream"
([s] {:pre [(s/valid? ::significant-string s)]
:post [(s/valid? ::byte-stream %)]}
(string->stream s "UTF-8"))
([s encoding]
(-> s
(.getBytes encoding)
(java.io.ByteArrayInputStream.))))
And in REPL I see what I expect to see from both generation and testing of the specs:
=> (instance? java.io.ByteArrayInputStream (string->stream "test"))
true
=> (s/valid? ::byte-stream (string->stream "0"))
true
=> (s/exercise-fn 'calais-response-processor.rdf-core/string->stream)
([("Y") #object[java.io.ByteArrayInputStream 0x57210dd7 "java.io.ByteArrayInputStream@57210dd7"]] [("d") #object[java.io.ByteArrayInputStream 0x7ec14113 "java.io.ByteArrayInputStream@7ec14113"]] [("5") #object[java.io.ByteArrayInputStream 0x1e85195b "java.io.ByteArrayInputStream@1e85195b"]] [("9c") #object[java.io.ByteArrayInputStream 0x3769ddef "java.io.ByteArrayInputStream@3769ddef"]] [("P0N") #object[java.io.ByteArrayInputStream 0x68793160 "java.io.ByteArrayInputStream@68793160"]] [("7tvN1") #object[java.io.ByteArrayInputStream 0x1cc43ca5 "java.io.ByteArrayInputStream@1cc43ca5"]] [("LjH4U") #object[java.io.ByteArrayInputStream 0x2a3da1a7 "java.io.ByteArrayInputStream@2a3da1a7"]] [("W") #object[java.io.ByteArrayInputStream 0x534287aa "java.io.ByteArrayInputStream@534287aa"]] [("x867VLr") #object[java.io.ByteArrayInputStream 0x72915e93 "java.io.ByteArrayInputStream@72915e93"]] [("moucN3vr") #object[java.io.ByteArrayInputStream 0x4f0d7570 "java.io.ByteArrayInputStream@4f0d7570"]])
But I don't understand why I'm seeing this from the test/check:
(stest/check 'calais-response-processor.rdf-core/string->stream)
({:spec #object[clojure.spec.alpha$fspec_impl$reify__2451 0x1acb0d46 "clojure.spec.alpha$fspec_impl$reify__2451@1acb0d46"], :clojure.spec.test.check/ret {:shrunk {:total-nodes-visited 4, :depth 3, :pass? false, :result #error {
:cause "Specification-based check failed"
:data {:clojure.spec.alpha/problems [{:path [:fn], :pred (clojure.core/fn [%] (clojure.core/instance? java.io.ByteArrayInputStream %)), :val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :via [], :in []}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$spec_impl$reify__1987 0x16a19b4c "clojure.spec.alpha$spec_impl$reify__1987@16a19b4c"], :clojure.spec.alpha/value {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.test.alpha/args ("0"), :clojure.spec.test.alpha/val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.alpha/failure :check-failed}
:via
[{:type clojure.lang.ExceptionInfo
:message "Specification-based check failed"
:data {:clojure.spec.alpha/problems [{:path [:fn], :pred (clojure.core/fn [%] (clojure.core/instance? java.io.ByteArrayInputStream %)), :val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :via [], :in []}], :clojure.spec.alpha/spec #object[clojure.spec.alpha$spec_impl$reify__1987 0x16a19b4c "clojure.spec.alpha$spec_impl$reify__1987@16a19b4c"], :clojure.spec.alpha/value {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.test.alpha/args ("0"), :clojure.spec.test.alpha/val {:args {:s "0"}, :ret #object[java.io.ByteArrayInputStream 0x7bee9d86 "java.io.ByteArrayInputStream@7bee9d86"]}, :clojure.spec.alpha/failure :check-failed}
:at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
:trace
[[clojure.core$ex_info invokeStatic "core.clj" 4739]
[clojure.core$ex_info invoke "core.clj" 4739]
...
...(lots more)
It feels like it's related to the generator fn composition although returned object looks "ok" to me right now.
:fn #(instance? java.io.ByteArrayInputStream %))
The problem is it looks like that :fn
spec expects only the function return value, when it's actually being invoked with a map containing the input and return values. Try this version instead:
:fn (fn [{:keys [args ret]}]
(instance? java.io.ByteArrayInputStream ret))
The :fn
spec should be a function that takes a map containing the function's input :args
and output :ret
value. It's meant to compare the function's output relative to its input.
In this example, the :fn
spec seems to be making the same assertion as your :ret
spec, and it doesn't look at the :args
so you may not want a :fn
spec here if there's no meaningful assertion to make between input/output — this would only assert the return value, redundantly.
And in REPL I see what I expect to see from both generation and testing of the specs
The reason you're only seeing the failure with check
is because none of those other calls are considering your function's :fn
spec e.g. s/exercise-fn
doesn't consider the :fn
spec.
I made some examples using :fn
specs here.