haskelltestingintegration-testingcabalblack-box-testing

Can I actually build and run an executable from the same package as part of a test suite?


It struck me that I do not really know of a way to black box test an executable packaged with Cabal.

With npm, for instance, I can run arbitrary shell commands, and I surely can wire it so that the necessary sources are transpiled and executed, and their side effects inspected.

Stack (as said here) builds the executables and publishes them in $PATH for the test suite, so I can easily run them.

But with Cabal, a test suite apparently cannot even depend on an executable, so there is no way to force the latter to be built. (Am I wrong about this?) And even then, I would have to know the path to the compiled binary.

How do I approach this problem?

The particulars of my situation are that an executable must extensively analyze the state of the system and branch accordingly, and I want to integration test that it does not forget to do so.

Note also that I am not at peace with running the relevant IO functions directly because I find it not integrative enough. Or, rather, I would like it to be possible to run the individual IO functions and also run the program as a whole. In my case, there are testing shell scripts in place already, but I would really like to "bake them in".


Solution

  • It turns out that there is a (slightly hacky) way to do this, at least for now, using the new(ish) build-tool-depends Cabal field. There has been some discussion (https://github.com/haskell/cabal/issues/5411, https://github.com/haskell/cabal/pull/4104#issuecomment-266838873) of build-tool-depends only being available at build-time, and having a separate field for executables that should be available when running a component. However, this separate run-time tool depends field doesn't exist yet. Luckily, it seems like Cabal (at least 2.1 and 2.2) completely doesn't draw this distinction: executables listed in build-tool-depends are actually available when cabal new-test runs a test suite. This means that you can use a pkg.cabal file that looks like this:

    name: pkg
    executable exe
      ...
    test-suite test
      ...
      build-tool-depends: pkg:exe
    

    And when you run the test suite, the executable will be built & on the path.