Clojure Test with Multiple Assertions and Reporting

I'm running into some issues with reporting on failures using the clojure.test testing framework.

Now, I understand that I can override some functions for different reporting so that it prints out to the console or wherever I want it to print to. I also understand that I can save this output to a file.

My issue is the following... when I declare a deftest like this example:

(deftest test1
  (is (= 1 1)
  (is (= 2 1))

This test will run and if I do something like (run-tests) or (test-var #'test1) it will return nil but print the failures.

I decided to override the :fail method for reporting, because what I want is a map of the failures like this: {"expected" (:expected m), "actual" (:actual m)} and this kinda sorta works if I were to just use the reporting function.

The problem is that when you run tests through the Clojure.test framework there are many macro's that get called and it doesn't behave it exactly how I want to.

My end goal is: running the tests, and if there are any failures, instead of printing them, save them to a map and return the map to me. If they all pass, then I don't care what it returns to me.

Is this even possible? I don't want to stop testing if a certain test fails, I just want it to be logged somewhere, preferably a map.

Sources:

Clojure test with multiple assertions

https://clojure.github.io/clojure/branch-1.1.x/clojure.test-api.html

https://groups.google.com/forum/#!topic/clojure/vCjso96wqps

Solution

I'm afraid there's no easy way how to do that. You could provide a custom implementation of clojure.test/report :fail defmethod and store the result in an atom but it's hard to propage the result to outer layers.

If you just use test-var then it's doable but note that test fixtures aren't executed in this case - see test-vars source:

(:use clojure.test)

(deftest failing
  (testing "fail me"
    (is (= 1 0))
    (is (= 2 1))
    (is (= 3 2))))

(def test-failures (atom []))

(defmethod report :fail [m]
  (swap! test-failures
         (fn [previous-failures current-failure]
           (conj previous-failures current-failure))
         {:test-var-str (testing-vars-str m)
          :expected (:expected m)
          :actual (:actual m)}))

(defmethod report :end-test-var [m]
  @test-failures)

(defn run-test-var [v]
  (reset! test-failures [])
  (test-var v))

;; in REPL:
(run-test-var #'failing)
;; =>
[{:test-var-str "(failing) (form-init4939336553149581727.clj:159)", :expected 1, :actual (0)}
 {:test-var-str "(failing) (form-init4939336553149581727.clj:160)", :expected 2, :actual (1)}
 {:test-var-str "(failing) (form-init4939336553149581727.clj:161)", :expected 3, :actual (2)}]

There's also defmethod report :end-test-ns but this one is not very useful because test-ns function returns @*report-counters*.