Signature created via CAPI cannot be verified using CNG, and vice versa

Background

• My application is using CAPI to sign and verify some data using RSA 2048.
• I'm trying to migrate to CNG, and it must be able to verify past signatures created with CAPI.

Problem

• Verification of CAPI signatures using NCryptVerifySignature() failed with NTE_BAD_SIGNATURE (0x80090006).
• I'm able to use CNG to verify CNG signatures.
• Tested verifying CNG signatures using CAPI (CryptVerifySignature()) but it failed with the same error.
• I'm able to use CAPI to verify CAPI-created signatures.

Test Code

https://godbolt.org/z/9Eb97v9xE

Overview of what it does:

1. Use CNG (MS_KEY_STORAGE_PROVIDER) to create a temporary RSA key pair.
2. Export the key pair as a LEGACY_RSAPRIVATE_BLOB for the next step.
3. Import the blob into CAPI. We now have a CNG handle and a CAPI handle to the key pair.
4. Use CNG to sign a blob of zeroes. Padding is set to PKCS1, hashing algorithm is set to SHA512.
5. Use CAPI to sign a blob of zeroes. CryptSignHash() does not allow callers to pass in the data to sign and instead takes in a HCRYPTHASH handle. Thus CryptSetHashParam() is used to directly set the hash value (all zeroes), which is what we want to sign. The hash algorithm is set to CALG_SHA_512.
6. Verify each signature using CAPI and CNG, given us a total of 4 tests. CNG-sign-CNG-verify and CAPI-sign-CAPI-verify passed, but the mixed API tests failed.

The code above uses NCryptXxx() functions. I had tried BCryptXxx() and it failed with the same error.

The only notable difference between my application and the test code is that the former imports the DER-encoded blobs of the public and private keys instead of generating a fresh pair every time.

CAPI or VerifyDataCapi() requires the signature in little endian order (here) , CNG or VerifyDataCng() in big endian order (here). To verify sigValCng with VerifyDataCapi(), the order must therefore be reversed. Likewise, this also applies to the verification of sigValCapi with VerifyDataCng().

Sample code:

std::reverse(std::begin(sigValCng), std::end(sigValCng)); // reverse sigValCng to little endian
printf("Sign with CNG, verify with CAPI\n");
VerifyDataCapi(hKeyCapi, hProv, hashVal, sigValCng); // Result: Test passed

std::reverse(std::begin(sigValCapi), std::end(sigValCapi)); // reverse sigValCapi to big endian
printf("Sign with CAPI, verify with CNG\n");
VerifyDataCng(hKeyCng, hashVal, hashValLen, sigValCapi); // Result: Test passed

https://godbolt.org/z/WKxxPKzYM