unit-testing

Can unit tests be environment-dependent?


I have this very generic question about unit test. Should we write environment dependent unit tests? The unit tests may depend on file system, a specific connection or presence of a file. I am not sure if there is a answer to it. But I found it takes time to decouple the unit tests from the environment. Does that mean it is ok to create environment dependent unit tests? Or it always points out that there is a design problem when we need to write environment dependent unit tests?


Solution

  • TL;DR

    Whether environment dependent tests are indeed a design flaw or not is strongly dependent on your application, programming language and specific problem. Keep in mind, that tests are never "complete" or "perfect" and try not to achieve perfection at the expense of more and more time. Using modern frameworks, making your tests re-usable and mocking your environment can take a lot of the pain out of testing.


    The longer answer

    IMHO, there's no perfect answer to this question. Unit Tests should be as generic and re-usable as possible without trying to cover environmental edge cases at any price.

    Example: Most environments should have an AMD64 processor nowadays (not regarding phones), so in most cases it is not needed to create unit tests for X32 or ARM environments, ...

    Your question is technology agnostic, so it's hard to tell, if the need for environment dependent unit tests is a design flaw. If you're using Java (see below), I'd rather think it is, using C++, it might not be...

    Some rules of thumb I can give from my experience (and therefore a little opinion-based):

    Follow the pareto principle

    Don't try to achieve 100% of what is possible in theory, but concentrate on the about 80% that are also likely to be encountered. Let's say, you expect your application to run on Windows Clients. So possibly your tests should cover the environmental characteristics of Windows 10 and 11. Older Windows versions, Linux, Mac OS, ... should not be needed.

    Or, if your application mainly has as Spanish audience, you should not care so much about how it can deal with chinese characters or canadian data protection rules.

    Exclude environmental specialties as much as sane and possible when coding your application

    If you can manage to make your application code mostly independent from your environment, the same will "automatically" apply to your tests. (Also, see the point about not re-inventing the wheel below)

    As you didn't ask for a specific programming language, I'll refer to Java here, but similar concepts exist in most modern languages.

    E. g., Java abstracts very much from your environment, so you can use File.separator, if you don't know if your application will be running on Windows (\ as separator) or Linux (/).

    Not using OS-specific APIs (like Java NI) avoids environment-related problems as well. If not dealing closely with hardware access, there shouldn't be any need to use them.

    Adapt your tests

    Like the "real" application, usually your tests are never "done". A new problem with your software arises? Fix it and add a specific test case for this kind of bug (trying to be quite generic). If it was a problem related to the 20% of cases or environments you did not consider before, be happy - now you've got a little hold of them as well.

    Package your test data with your application

    Referring to Java again, you could place your test files in the resources folder and load them dynamically while testing - not need to bother the real file system of the testing machine.

    Also, if you're using a database, you could spin up an in-memory instance (like H2) during testing. This will not exactly mirror your production database system, but makes testing very easy without the danger of breaking something. Anyway, database abstraction is also very sophisticated nowadays, so despite of some edge cases, you should not see any difference in behaviour if you're using a good database abstraction layer (which itself in turn make you less environment-dependent).

    Make your tests re-usable

    Using a tool like Maven or Gradle, you can make your tests run on a bigger variety of environments, if they are packaged as just pointed out. The need for a specific test environment will decrease.

    Mocking is your friend

    As with the "fake" database mentioned before, you can mock a lot of things. There are frameworks like Mockito allowing white box and black box testing with mocked objects. "Helpers" like WireMock allow you to even mock the answer you get from an external service. And there are a lot more possibilities out there!

    Don't re-invent the wheel

    If you use one of todays advanced application frameworks (like Spring, Symfony, Boost, ...) they'll do a lot of the work for you if you make proper use of them. You'll have to read the docs and burrow into them, but then you will realize how much easier they can make your life. This does not apply to frameworks only, but also to libraries, services, components, ... And the good thing is, all of that will also come into place when it's about testing! Possibly most of your environmental dependencies might vanish if you use a good software foundation.