Spacemacs Cider test does not fail

When I run a test in Spacemacs with Clojure mode via <SPC> m t t it does not show a failure, even when the test clearly fails. See:

1 is not equal to 2, but still there are 0 test failures.

How can I make the test fail?

There is an issue in your test: it lacks comparison operator. The correct version is:

(deftest test-exercise-1-4
  (testing "count-anywhere"
    (is (= 2 1))))

is macro has following definition (edited source for brevity):

(defmacro is
  "Generic assertion macro.  'form' is any predicate test.
  'msg' is an optional message to attach to the assertion.

  Example: (is (= 4 (+ 2 2)) \"Two plus two should be 4\")

  ([form] `(is ~form nil))
  ([form msg] `(try-expr ~msg ~form)))

As you can see with (is 2 1) you are calling the second arity where form is 2 and the message for assertion is 1. As 2 is truthy it makes the assertion to pass:

(is 2)
;; => 2

(is false)

FAIL in () (boot.user5045373352931487641.clj:1)
expected: false
  actual: false
;; => false