My problem concern the usage of an hsm with Java (openjdk 11.0.12). Hsm should be use for signature purpose, by SHA512 RSA algorithm. I could be wrong in a lot of the following sentences, I'm totally newbie with HSM & co, so I apologize in advance.
For what I've understand there are three kinds of approaches:
1- Using SUNPKCS11 provider
2- Using vendor lib (hsm is shipped with a couple of jar, in my case nCipher is shipped with nCipherKM.jar, which should be vendor provider.)
3- openssl (we have some software in c already doing this, I prefer to avoid)
The usage of vendor lib it's really easy, at least until the Get info call, which send an Unknown Parameter to HardServer, causing an unmarshable exception. This is difficult to debug, communication protocol isn't documented. Right now I've put this solution aside.
In any case I prefer the SUNPKCS11 solution, it doesn't work out of the box for me, but it was simple to debug and analyze. And should be a standard.
In this case i'm using European DSS library to interface with PKCS11Provider, making things a little simpler for me to configure and implement.
The problem occurs during SunPKCS11 (vanilla) initialization.
At some point it calls a method "P11Keystore.mapLabels()" that match, according to code and Oracle documentation, all private key handlers (CKA_PRIVATE_KEY) coming from that slot with certificate handlers (CKO_CERTIFICATE), looking for matching between cka_id, in order to build a software in memory keystore with aliases map containing the CKA_LABEL attributes. (Private key is unextractable so access is read only https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#KeyStoreRestrictions)
In signature initialization this private key entry is used to fetch from HSM (by some key attributes that I don't have) the private key handler.
The problem is that my hsm nCipher doesn't expose any object for CKO_CERTIFICATE, so the match returns 0 result and my software keystore is empty.
When I try to extract the private key handler from keystore I obtain nothing and I cannot initialize Signature object.
My predecessor manually wrapped the private key attributes inside a local jks, and rewrote a new provider in order to load certificate from file and not from HSM/PKCS11.
I dislike this solution, I don't want my application to have configuration depending on HSM certificate.. it's HSM certificate job to handle those keys, not mine.
Instead, I wrote another provider to fetch and use directly the private key handler, from CKA_PRIVATE_KEY, using a preconfigured CKA_LABEL, bypassing the certificate match. And it works.
However I dislike this solution too, it means more maintenance costs for a standard protocol, and the jar must be signed each time, which for me is a nuisance.
I have the feeling that I am approaching the problem from the wrong side, maybe because I'm a noob in the matter.
Explanation are over, so here my questions: 1- Am I wrong to claim that CKO_CERTIFICATE is a prerequisite for SunPKCS11? 1- Could/Should HSM expose CKO_CERTIFICATE object without malevolent side effects? 2- Is this missing object a limitation of nCipher HSM or, probably, a configuration missing during installation? (It works even without so it's a java prerequisite more than an HSM missing) 3- If the CKO_CERTIFICATE cannot be installed and exposed: Is it ok to implement our own provider to obtain the workaround, or could exist a better way to get it working?
Sorry for my English, I'm not a native one. Thanks to those who have come to read up to here and who will answer.
..almost a month later..
I've finished my application, now I know a lot more about the argument, it works with following modes:
I've tried to extend Bouncy Castle Fips provider, instead of SunPKCS11, without any luck.
In the end I think is not possible to accomplish what I need for, the problem is in the server configuration, which is not solvable from a client software. Anyway I'll fix server configuration, adopting the first working case, dropping custom PKCS11 solution, keeping it just for academic purpose.