opensslsignpem

openssl CMS_verify() returns "invalid digest" after updating to OpenSSL 3

I am recomping a utility which downloads a signed message from a server, verifies the signature and then decodes the data in the message if signature verification succeeded. This works fine when linked with OpenSSL 1.0.2k (CentOS 7) and 1.1.1n (Debian 11), but not with OpenSSL 3.0.1 (Rocky Linux 9), here I get an error :error:03000098:digital envelope routines:do_sigver_init:invalid digest:crypto/evp/m_sigver.c:343:

Here's the gist of the code that fails with OpenSSL 3 (error handling removed):

X509_STORE *st = X509_STORE_new();
X509 *cacert = rootcert(); /* retrieves signing certificate */
X509_STORE_add_cert(st, cacert);
CMS_ContentInfo *cms = PEM_read_bio_CMS(in, NULL, NULL, NULL);
CMS_verify(cms, NULL, st, NULL, out, 0);

I also tried running the command-line equivalent:

openssl cms -verify -in message.pem -inform PEM -CAfile rootcert.pem

With OpenSSL 1.0 and 1.1, this outputs the decoded message as expected.

When run with OpenSSL 3.0, I get the same "invalid digest" error as mentioned above:

$ openssl cms -verify -in message.pem -inform PEM -CAfile cacert.pem 
CMS Verification failure
800BA77A767F0000:error:03000098:digital envelope routines:do_sigver_init:invalid digest:crypto/evp/m_sigver.c:343:

What am I doing wrong?

The message comes from anGo server, which basically using the code in the example for type fullsailor/pkcs7.SignedData described here, except that it does not detach the signature (again, error handling removed):

signedData, err := pkcs7.NewSignedData(message)
signedData.AddSigner(publicCert, privateKey, pkcs7.SignerInfoConfig{})
pem.Encode(out, &pem.Block{Type: "CMS", Bytes: sign})
Solution

  • OpenSSL 3 discourages RSA-1 signatures, which are the type generated by the https://github.com/fullsailor/pkcs7 used in the example above. As mentioned in the comment from @Matt Caswell, switching OpenSSL 3 to legacy mode does make it work.

    Replacing the signing library with the fork at https://github.com/mozilla-services/pkcs7 and adding the the line below to request a SHA-256 signature gives me something that OpenSSL 3 allows with the default settings:

    signedData.SetDigestAlgorithm(pkcs7.OIDDigestAlgorithmSHA256)