asdf:test-system from a Makefile doesn't return an error return code

My unit tests run fine, but the make test target won't return an error code when the tests fail (echo $? is 0) (it does when there is a lisp error).

This is not good for a CI system.

My main system definition ends with in-order-to:

  :in-order-to ((test-op (test-op "mysystem/test"))))

My test system has a perform one:

  :perform (test-op (o s)
                    (uiop:symbol-call '#:mysestem.test '#:test-all)))

where test-all is a function that runs Fiveam test suites with run! 'test-suite.

The make target:

test:
  # (with proper indentation…)
  $(LISP) --non-interactive \
    --eval "(asdf:load-asd \""$(ASD_PATH)"\")" \
    --eval "(ql:quickload :mysystem/test)" \
    --eval "(asdf:test-system \"mysystem\")"

Note that follownig ASDF's best practices in doing

                    (symbol-call :fiveam #':run! :test-suite)))

errors out with "The function :run! is undefined."

How do we do this ?

Thank you.

Solution

The problem is, as @jskiiski points out, that asdf:test-system returns true regardless of what the test-op returned.

I'm using rove instead of fiveam, and rove:run actually does return nil if the tests failed. So my workaround is to call it directly (instead of asdf:test-system), and use uiop:quit to get the right return code:

sbcl --non-interactive \
     --eval '(ql:quickload :mysystem)' \
     --eval '(asdf:load-system :rove)' \
     --eval '(uiop:quit (if (rove:run "mysystem/tests") 0 1))'

I decided against trying to fiddle that logic into test-op because I use asdf:test-system interactively most of the time.

Can't think of any solution that doesn't feel like a workaround though. asdf:test-system just passing along whatever test-op returns (or something to that effect) seems like the only proper fix.