I am trying to sign a byte string using the c_sign function from pkcs and I am trying to verify it using RSA.verify() function in ASP.net, but for some reason its failing.
I am using 2048 bit RSA keys and I am trying to create a RSA SHA384 signature and I have tried using both CKM_RSA_PKCS and CKM_SHA384_RSA_PKCS and I have tried to verify them using c_verify and its giving success but when I tried to verify the output from .net its failing. Is it because the regular signature creation process is different from the PKCS c_sign process ?
An RSA signature needs padding (usually PKCS#1 v1.5 or PSS), and unless your input is guaranteed to be short, a hash algorithm (like SHA384).
CKM_RSA_PKCS uses PKCS#1 v1.5 without hashing, while CKM_SHA384_RSA_PKCS adds SHA384. I could not find which RSA.verify function you mean in ASP.net, but you need to ensure that both sides (1) are hashing the input exactly once, and (2) are using the same padding scheme.