[SOLVED] RSA OAEP Encryption with SHA-256 fails while with SHA-1 is OK

RSA OAEP Encryption with SHA-256 fails while with SHA-1 is OK

I'm using Pkcs11Interop Library and trying to test encryption and decryption with RSA_PKCS_OAEP mechanism.

CK_RSA_PKCS_OAEP_PARAMS p = new CK_RSA_PKCS_OAEP_PARAMS();
p.HashAlg = (uint)CKM.CKM_SHA_1;
p.Mgf = (uint)CKG.CKG_MGF1_SHA1;
p.Source = (uint)CKZ.CKZ_DATA_SPECIFIED;
p.SourceData = IntPtr.Zero;
p.SourceDataLen = 0;

CK_MECHANISM mech = CkmUtils.CreateMechanism(CKM.CKM_RSA_PKCS_OAEP, p);

Everything is OK with the above mechanism but if I change the hash algorithm to SHA-256 like below:

CK_RSA_PKCS_OAEP_PARAMS p = new CK_RSA_PKCS_OAEP_PARAMS();
p.HashAlg = (uint)CKM.CKM_SHA256;
p.Mgf = (uint)CKG.CKG_MGF1_SHA256;
p.Source = (uint)CKZ.CKZ_DATA_SPECIFIED;
p.SourceData = IntPtr.Zero;
p.SourceDataLen = 0;

CK_MECHANISM mech = CkmUtils.CreateMechanism(CKM.CKM_RSA_PKCS_OAEP, p);

Then I get CKR_ARGUMENTS_BAD exception. I have been searching and debugging for a while but found nothing.

Solution

I had the same problem with Luna HSM (but was given CKR_MECHANISM_PARAM_INVALID).

That version of HSM simply did not support OAEP with SHA-256 and firmware upgrade was needed. After firmware upgrade it worked without any problems. Check if your device supports this variant.

Your code seems ok, I used (in java):

CK_RSA_PKCS_OAEP_PARAMS mechanismParams = new CK_RSA_PKCS_OAEP_PARAMS(
    CKM.SHA_1,
    CKG.MGF1_SHA1,
    new CK_RSA_PKCS_OAEP_SOURCE_TYPE(CKZ.DATA_SPECIFIED.longValue())
    , null, 0
);

and

CK_RSA_PKCS_OAEP_PARAMS mechanismParams = new CK_RSA_PKCS_OAEP_PARAMS(
    CKM.SHA256,
    CKG.MGF1_SHA256,
    new CK_RSA_PKCS_OAEP_SOURCE_TYPE(CKZ.DATA_SPECIFIED.longValue())
    , null, 0
);

Good luck!