[SOLVED] How should I solve the problem with the xmlsec1 library that I compiled myself for iOS - transformCtx.first is NULL after xmlSecTransformCtxPrepare()

How should I solve the problem with the xmlsec1 library that I compiled myself for iOS - transformCtx.first is NULL after xmlSecTransformCtxPrepare()

I have built xmlsec1 on my own for iOS and when I test it it seems as if my build is not quite successful :)

I have the code that worked fine on Mac OS with xmlsec I got via macports.

But since there is no iOS version I can get with cocoapods I decided to build it myself - using openssl for crypto. Now I am testing it with a certificate that is valid and works fine with MacOS version

xmlSecTransformCtx     transformCtx;
xmlSecTransformPtr     signMethod;
xmlSecByte            *dataPtr = (xmlSecByte *)inStr;
xmlSecDSigCtxPtr       dsigCtx = NULL;

...

memset (&transformCtx, 0, sizeof(xmlSecTransformCtx));  // was  FillChar (transformCtx, SizeOf(transformCtx), #0);

xmlSecTransformCtxInitialize (&transformCtx);

dsigCtx = xmlSecDSigCtxCreate (NULL);

if (!dsigCtx)  {
  // I'm checking all the errors and there are none in the code below until the last line
  ...
}

dsigCtx->signKey = xmlSecCryptoAppKeyLoad (cert_file, xmlSecKeyDataFormatPkcs12, password, NULL, NULL);   // No errors

signMethod = xmlSecTransformCtxCreateAndAppend (&transformCtx, xmlSecTransformRsaSha1Id);   // signMethod ok

signMethod->operation = xmlSecTransformOperationSign;

errCode = xmlSecTransformSetKey (signMethod, dsigCtx->signKey);  // no errCode

errCode = xmlSecTransformCtxPrepare (&transformCtx, xmlSecTransformDataTypeBin);

if (errCode < 0)  {
  // AGAIN no error code
}

if (( transformCtx.first ) == NULL)  fprintf (stderr, "NULL!");

// It is NULL, xmlSecTransformDefaultPushBin() fails

errCode = xmlSecTransformDefaultPushBin (transformCtx.first, dataPtr, (xmlSecSize)dataSize, 1, &transformCtx);

Because transformCtx.first is NULL xmlSecTransformDefaultPushBin() fails.

Any idea what could have gone wrong in my build and how should I start solving this problem.

EDIT - seems to be a problem with struct alignment.

Even though xmlSecTransformCtx is 128 bytes in my app and inside the library, fields are mismathed. transformCtx.first is NULL in my own code but inside of xmlSecTransformCtxPrepare() is not. Strange.

Solution

In the end it turnes out I had defined XMLSEC_NO_SIZE_T in my app becauese I created it from an example I used 10 years ago and the library was compiled without XMLSEC_NO_SIZE_T.

This changes a lot:

#ifdef XMLSEC_NO_SIZE_T
#define xmlSecSize                              unsigned int
#else  /* XMLSEC_NO_SIZE_T */
#define xmlSecSize                              size_t
#endif /* XMLSEC_NO_SIZE_T */

So the size of internal structure was different and gdb was showing me only the sizeof for structures as they were in app even when I was stepping inside the library.