Perl XS: create and return array of strings (char*) taken from calling a C function or undef on failure

I have Perl XS code which calls a function from an external C library which returns char ** (array of strings).

The XS code will eventually return back to Perl an array ref with all the string results in there. Or undef on failure.

I have 2 problems:

1. On program exit I get a core dump with messages about memory corruption, double free etc. (e.g. double free or corruption (fasttop)).
2. How to return an undef value from XS sub denoting that something went wrong (not an empty array)?

Additionally, if anyone can confirm that I am handling correctly the cases where strings from Perl into the C function are utf8-encoded (e.g. the input filename) or the results back from the C function (which may contain utf8 strings) are sent back to Perl OK.

Here's my code (which is modelled after https://stackoverflow.com/a/46719397/385390 If I got that correctly, example #1):

AV *
decode(infilename_SV)
    SV *infilename_SV
  PREINIT:
    char *infilename;
    STRLEN infilename_len;
    char **results;
    size_t results_sz;
    char *aresult;
    size_t I;
    SV **aresultPP;
    char *dummy;
    STRLEN dummy_len;
  CODE:
    infilename = SvPVbyte(infilename_SV, infilename_len)
    // call C function
    results = myfunc(infilename, &results_sz);
    if( results == NULL ){
      printf("error!");
      // HOW TO return undef (and not an empty array?)
    }
    // create a Perl array to be returned
    RETVAL = (AV*)sv_2mortal((SV*)newAV());
    for(I=0;I<results_sz;I++){
      results_sz = strlen(results[I]);
      // create a new Perl string and copy this result
      aresult = newSVpv(results[I], 0);
      av_push(RETVAL, aresult);
      // free results as returned by C call
      free(results[I]);
    }
    // free results as returned by C call
    free(results);
    // debug print results
    for(I=0;I<results_sz;I++){
      aresultPP = av_fetch((AV *)RETVAL, I, 0);
      dummy = SvPVbyte(*apayloadPP, dummy_len);
      printf("result: %s\n", dummy);
    }
  OUTPUT:
     RETVAL

On program exit I get a core dump with messages about memory corruption, double free etc. (e.g. double free or corruption (fasttop)).

This was probably because you overwrote the loop variable results_sz inside the for causing undefined behavior.

How to return an undef value from XS sub denoting that something went wrong (not an empty array)?

You can return &PL_sv_undef to signal an undefined value, see perlxs for more information. For example like this:

SV *
decode(infilename_SV)
    SV *infilename_SV
  PREINIT:
    char *infilename;
    STRLEN infilename_len;
    char **results;
    size_t results_sz;
    char *aresult;
    size_t I;
  CODE:
    infilename = SvPVbyte(infilename_SV, infilename_len);
    results = myfunc(infilename, &results_sz);
    if( results == NULL ){
      RETVAL = &PL_sv_undef;
    }
    else {
       AV *av = newAV();
       for(I=0; I < results_sz; I++){
          aresult = newSVpv(results[I], 0);
          av_push(av, aresult);
          free(results[I]);
       }
       free(results);
       RETVAL = sv_2mortal(newRV_noinc((SV*)av));
    }
  OUTPUT:
     RETVAL

if anyone can confirm that I am handling correctly the cases where strings from Perl into the C function are utf8-encoded (e.g. the input filename)

To pass a Perl UTF-8 string to the C-function as an UTF-8 encoded character string you can use SvPVutf8() instead of SvPVbyte(), see perlguts for more information. Example:

infilename = SvPVutf8(infilename_SV, infilename_len);

or the results back from the C function (which may contain utf8 strings) are sent back to Perl

You can use newSVpvn_flags() instead of newSVpvn() to convert an UTF-8 encoded C-string to a Perl string. For example:

aresult = newSVpvn_flags(results[I], strlen(results[I]), SVf_UTF8);