I am trying to dynamically generate a bunch of test cases.
Basically I want to run the same tests several times with different values. Maybe there's a better way to do that, if so let me know, I haven't found it yet.
I found make-test-suite which says you can pass it a list of test-case instances.
So I made a for/fold loop which collects the test cases generated for each set of values into a flat list.
What I don't understand is it seems to successfully run all the tests while also raising an exception:
tests->test-suite-action received #<void> in list of tests (#<void> #<void> #<void> #<void> #<void> #<void> #<void> #<void>), which is not a test.
context...:
/usr/share/racket/pkgs/rackunit-lib/rackunit/private/test-suite.rkt:139:7
/usr/share/racket/pkgs/rackunit-lib/rackunit/private/test-suite.rkt:136:2
/usr/share/racket/pkgs/rackunit-lib/rackunit/private/test-suite.rkt:61:0: apply-test-suite
/usr/share/racket/pkgs/rackunit-lib/rackunit/text-ui.rkt:91:0: run-tests
"/write-thru-hash/tests.rkt": [running body]
temp35_0
for-loop
run-module-instance!
/usr/share/racket/pkgs/compiler-lib/compiler/commands/test.rkt:179:16
...
1 1 write-thru-hash/tests.rkt
8 tests passed
rkt-write-thru-hash_tests_1 exited with code 1
I put a writeln in each of my test cases and I see those lines printed out where I abbreviated with ... above. So I know the tests are actually running.
(The tests also ran and worked fine before I multiplied them with the for/fold loop and just built them in the body of test-suite)
My tests.rkt code looks like:
(define test-cases-list
(for/fold ([test-cases (list)])
([db-type (list 'sqlite 'postgres)])
(append test-cases
(list
(test-case/fixture "basic operations" ... )
(test-case/fixture "complex serializable keys and values" ... )
(test-case/fixture "custom table-name" ... )
(test-case/fixture "use initial src-hash" ... )))))
(define db-tests
(make-test-suite "db-tests" test-cases-list))
(I'm using the fixture library https://docs.racket-lang.org/fixture/)
UPDATE:
Actually I think the writeln in my test cases are printing at definition time ... i.e. they are running too early because they are outside of the test-suite context.
I guess the tail check in each test-case returns #<void>, so I have filled a list with the results of the tests (voids) rather than the test cases themselves and given that to the test suite, hence the error.
But then I don't see how to actually use make-test-suite...?
Eventually found a simple way to do this.
My problem was in trying to have several variations of my test cases all dynamically generated into one test-suite. I did not find a way to get that to work, I suspect I would have to write a macro.
Instead the simple way is to dynamically define multiple test-suite instances and then loop over a list of them to run them:
(define db-test-suites
(for/list ([db-type (list 'sqlite 'postgres)]
[db-conn-fixture (list sqlite-connection-delete-after-use
postgres-connection)])
(define suite-name (format "db-tests: ~a" db-type))
(test-suite suite-name
(test-case/fixture "basic operations"
#:fixture db-conn-fixture
(define db-conn (fixture-value db-conn-fixture))
...)
(test-case/fixture "complex serializable keys and values"
#:fixture db-conn-fixture
(define db-conn (fixture-value db-conn-fixture))
...))))
(for ([current-test-suite db-test-suites])
(run-tests current-test-suite))
UPDATE:
I also found a way to actually use make-test-suite:
(define all-tests (make-test-suite "db-test-suites" db-test-suites))
(run-tests all-tests)
This takes advantage of the fact that a test-suite can contain other nested test-suites. I think this form is nicer (a single run-tests call) than the one above (multiple run-tests calls in a for loop).
All my problems in the original question were in trying to pre-define a bunch of test-case instances outside of a test-suite container. But here we pass make-test-suite a list of test-suite instances, allowing the tests within them to defer running until we later call run-tests.