ctestingparameterized-unit-testparameterized-testsc-criterion

Can't parameterize tests by strings


I'm trying to write a parameterized test based on some string values using the Criterion framework. As an MRE, and following the example shown in the docs, I wrote the following:

#include <criterion/criterion.h>
#include <criterion/new/assert.h>
#include <criterion/parameterized.h>
#include <stdio.h>

ParameterizedTestParameters(test_suite, test_str) {
    static char* strings[] = {"Foo", "Bar"};
    size_t strings_count = sizeof(strings)/sizeof(char*);

    return cr_make_param_array(char*, strings, strings_count);
}

ParameterizedTest(char** pstring, test_suite, test_str) {
    printf("%s\n", *pstring);
}

However, when I run the test, it crashes with the following output:

[----] tests/string_aparams.c:13: Unexpected signal caught below this line!
[FAIL] test_suite::test_str: CRASH!
[----] tests/string_aparams.c:13: Unexpected signal caught below this line!
[FAIL] test_suite::test_str: CRASH!

But when I tried writing a similar test based on an array of int parameters, it worked fine:

ParameterizedTestParameters(test_suite, test_int) {
    static int numbers[] = {1, 2};
    size_t numbers_count = sizeof(numbers)/sizeof(int);

    return cr_make_param_array(int, numbers, numbers_count);
}

ParameterizedTest(int* pnum, test_suite, test_int) {
    printf("%d\n", *pnum);
}
[====] Running 2 tests from test_suite:
[RUN ] test_suite::test_int
[RUN ] test_suite::test_int
1
2
[PASS] test_suite::test_int: (0.00s)
[PASS] test_suite::test_int: (0.00s)

What could be the problem with first snippet of code?

UPDATE:

After re-reading the docs a couple of times, the following passage from the same page made me think:

Any dynamic memory allocation done from a ParameterizedTestParameter function must be done with cr_malloc, cr_calloc, or cr_realloc

Perhabs using "Foo" and "Bar" inside {"Foo", "Bar"} is dynamically allocating space for the two strings and I should instead use cr_malloc to allocate space for them. I'm not sure if that is the case, but this made me try another approach:

#include <criterion/criterion.h>
#include <criterion/new/assert.h>
#include <criterion/parameterized.h>
#include <stdio.h>
#include <string.h>

#define STRINGS_COUNT 2 
#define STRINGS (char*[]){"Foo", "Bar"}

ParameterizedTestParameters(test_suite, test_str) {
    static char* strings[STRINGS_COUNT];
    
    for(size_t k = 0; k < STRINGS_COUNT; k++) {
        strings[k] = cr_malloc(strlen(STRINGS[k]) + 1);
        strcpy(strings[k], STRINGS[k]);
    }

    return cr_make_param_array(char*, strings, STRINGS_COUNT);
}

ParameterizedTest(char* *pstring, test_suite, test_str) {
    printf("%s\n", *pstring);
}

When running the above test, the following output is displayed:

[RUN ] test_suite::test_str
Foo
[PASS] test_suite::test_str: (0.00s)
[RUN ] test_suite::test_str
Bar

Is my assumption that using the initializer {"Foo", "Bar"} dynamically allocate space for the two strings correct? And, if not, why would the first test crash while the third succeed?


Solution

  • The crash occurs because Criterion runs tests in their own child process, and the string address isn't valid in the child process.

    As you found out, using cr_malloc is the recommended fix. To avoid memory leaks, you should add a bit of code with corresponding calls of cr_free.

    See https://github.com/Snaipe/Criterion/issues/258 for details, "Crash with strings inside ParameterizedTestParameters":

    This is due to the side-effect that tests are run in their own child processes, and the address of "criterion" becomes invalid. See this issue for more information.

    TL;DR: use cr_malloc to allocate your strings in ParameterizedTestParameters to use them in ParameterizedTest.

    The linked issue is https://github.com/Snaipe/Criterion/issues/208:

    Criterion is behaving correctly: it transfers your parameters verbatim as they are specified in ParameterizedTestParameters to your test function. Unfortunately, this means passing the address of "a\nb\nc\n" as-is, to the new process, which due to ASLR will have a different address space layout. [...] or use cr_malloc to notify criterion that the object have to stay at the same address

    The section Freeing dynamically allocated parameter fields in the Criterion documentation explains how to avoid memory leaks. It doesn't specifically disallow simply calling cr_free in your ParameterizedTest code when you're sure you won't need this item anymore, so maybe that's OK as well.