cvalidationtestingstandards-compliancesuite

Why C compiler validation suites themselves may use the C standard library if the C standard library is actually under test?


How it is possible for C compiler validation suites to use any feature of the C standard library if the C standard library is actually under test and may be buggy (which invalidates the validation suite itself)?

For example, the suite may use itself memset, fopen, etc., but memset, fopen, etc. are actually under test and may be buggy.

Confused. Clarification / experience is needed.

UPD. Clarified after examining suite's source code. The suite uses two compilers:

  1. Reference compiler is used to build the suite itself.
  2. IUT (Implementation Under Test) compiler is used to run tests on it.

Solution

  • Yes, testing implementation X using implementation X is not correct (with exceptions, see below). In my particular case the situation is clarified after examining suite's source code. The suite uses two compilers:

    1. Reference compiler is used to build the suite itself.
    2. IUT (Implementation Under Test) compiler is used to run tests on it.

    Exceptions. As user Steve Summit mentioned:

    A good test suite may consist of many stages or layers. First you test A against something fundamental. Then, once you know A is correct, you can use A as part of your test to validate B. Then, you can use A and B to validate C, etc.

    This is viable approach. Hover, since the first (upper) layer cannot be tested (for example the correctness of the == operator), we take as an axiom that the first layer is error free.

    Also continue quoting user Steve Summit:

    Sometimes it's even important to test your tests.

    I'd say that it is always important to test the correctness of the tests themselves.