c++opensslrsadigital-signature

signature generated using openssl C++ API does not match with same code in python

I have a python code which generates signature based on hash string as data. It uses cryptography library to calculate singature based on hash. It uses private key file .pem which contains private key and public key as well. It uses serialization.load_pem_private_key() API to get the private key by reading .pem file. Then using this private key it generates signature using following method of cryptography in python.

privateKey.sign(hashData,
     padding.PSS(mgf=padding.MGF1(hashes.SHA256()), salt_length=32),
     utils.Prehashed(hashes.SHA256()))

It always generates a new signature if even hashData is same which is input here. It then verifies this generated signature by fetching the public key from same .pem file and then calling verify API as follows

publicKey.verify(
                generated_signature,
                hashData,
                padding.PSS(
                    mgf=padding.MGF1(hashes.SHA256()),
                    salt_length=32
                ),
                utils.Prehashed(hashes.SHA256())
            )

Verification succeeds in python. I tried to mimic same logic in C++ using OpenSSL API of version 3.4.0. The signature generation is successful but when I try to verify it using this python based tool the verification fails by error

failed to verify with private key from key file 'key.pem'.

Following is my sample C++ code written to mimic same logic

#define _CRT_SECURE_NO_WARNINGS 
#include "openssl/pem.h"
#include "openssl/err.h"
#include "openssl/evp.h"
#include "openssl/rsa.h"
#include "openssl/sha.h"
#include <iostream>
#include <fstream>
#include <sstream>
#include <iomanip>
#include <vector>
#include "ms/applink.c"


void handleOpenSSLErrors() 
{
    ERR_print_errors_fp(stderr);
    abort();
}

// Load RSA private key from a PEM file
EVP_PKEY* loadPrivateKey(const std::string& filePath) 
{
    FILE* keyFile = nullptr;
    errno_t r_code = fopen_s(&keyFile, filePath.c_str(), "rb");
    if (!keyFile) {
        std::cerr << "Unable to open private key file: " << filePath << std::endl;
        return nullptr;
    }
    EVP_PKEY* pkey = PEM_read_PrivateKey(keyFile, nullptr, nullptr, nullptr);
    fclose(keyFile);

    if (!pkey) 
    {
        std::cerr << "Unable to read private key from file: " << filePath << std::endl;
    }

    int keyType = EVP_PKEY_base_id(pkey);

    switch (keyType) {
    case EVP_PKEY_RSA:
        std::cout << "Key Type: RSA" << std::endl;
        break;
    case EVP_PKEY_EC:
        std::cout << "Key Type: EC (Elliptic Curve)" << std::endl;
        break;
    case EVP_PKEY_DSA:
        std::cout << "Key Type: DSA" << std::endl;
        break;
    default:
        std::cout << "Key Type: Unknown" << std::endl;
        break;
    }

    return pkey;
}

std::vector<unsigned char> signData(EVP_PKEY* privateKey, const std::vector<unsigned char>& hashData) {
    EVP_MD_CTX* ctx = EVP_MD_CTX_new();
    if (!ctx) handleOpenSSLErrors();

    if (EVP_DigestSignInit(ctx, nullptr, EVP_sha256(), nullptr, privateKey) <= 0) {
        handleOpenSSLErrors();
    }

    EVP_PKEY_CTX_set_rsa_padding(EVP_MD_CTX_get_pkey_ctx(ctx), RSA_PKCS1_PSS_PADDING);
    EVP_PKEY_CTX_set_rsa_mgf1_md(EVP_MD_CTX_get_pkey_ctx(ctx), EVP_sha256());
    EVP_PKEY_CTX_set_rsa_pss_saltlen(EVP_MD_CTX_get_pkey_ctx(ctx), -2);
    
    size_t sigLen = 0;
    if (EVP_DigestSign(ctx, nullptr, &sigLen, hashData.data(), hashData.size()) <= 0) {
        handleOpenSSLErrors();
    }

    std::vector<unsigned char> signature(sigLen);
    if (EVP_DigestSign(ctx, signature.data(), &sigLen, hashData.data(), hashData.size()) <= 0) {
        handleOpenSSLErrors();
    }

    EVP_MD_CTX_free(ctx);
    return signature;
}

std::string vectorToHexString(const std::vector<unsigned char>& bytes)
{
    std::ostringstream hexStream;

    // Convert each byte to a two-character hexadecimal representation
    for (unsigned char byte : bytes) {
        hexStream << std::hex << std::setw(2) << std::setfill('0') << static_cast<int>(byte);
    }

    return hexStream.str();
}

std::vector<unsigned char> hexStringToBinary(const std::string& hex) {
    if (hex.length() % 2 != 0) {
        throw std::invalid_argument("Hex string length must be even");
    }

    std::vector<unsigned char> binary;
    binary.reserve(hex.length() / 2);

    for (size_t i = 0; i < hex.length(); i += 2) {
        std::string byteString = hex.substr(i, 2); // Get two characters
        unsigned char byte = static_cast<unsigned char>(std::stoi(byteString, nullptr, 16));
        binary.push_back(byte);
    }

    return binary;
}

int main()
{
    const std::string privateKeyPath = "private_key.pem";

    // Example hash data
    std::string data = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855";
    
    std::vector<unsigned char> hashBinary = hexStringToBinary(data);
    for (unsigned char byte : hashBinary)
    {
        std::cout << std::hex << std::setw(2) << std::setfill('0') << static_cast<int>(byte);
    }
    // Verify size
    if (hashBinary.size() != 32) {
        std::cerr << "Invalid hash size! Expected 32 bytes (SHA-256)." << std::endl;
        return 1;
    }
    // Load keys
    EVP_PKEY* privateKey = loadPrivateKey(privateKeyPath);
    if (!privateKey) return 1;

    if (EVP_PKEY_base_id(privateKey) == EVP_PKEY_RSA)
    {
        std::cout << "RSA key loaded successfully." << std::endl;
    }
    else 
    {
        std::cerr << "Invalid key type! RSA key required." << std::endl;
        EVP_PKEY_free(privateKey);
        return 1;
    }
    // Generate signature
    std::vector<unsigned char> signature = signData(privateKey, hashBinary);
    std::cout << "Signature generated successfully." << std::endl;
    for (unsigned char byte : signature)
    {
        std::cout << std::hex << std::setw(2) << std::setfill('0') << static_cast<int>(byte);
    }
    
    // Clean up
    EVP_PKEY_free(privateKey);

    return 0;
}

in C++ code if I load public key and try to verify it using OpenSSL API then verification succeeds. But this generated signature is failed in verification when try to verify it using Python based tool !

How should I debug it and how to resolve this issue ? Am I missing something in C++ Code ?

Solution

  • The EVP_DigestSignXXX() functions you use in signData() hash implicitly, so that the C code hashes twice overall. You may have assumed that this is not the case, as there are also the EVP_SignXXX() functions. But these also hash implicitly (please note, that both are similar, but the former are newer and should be used in newer applications, see here).

    What you need are the EVP_PKEY_signXXX() functions, which do not implicitly hash, e.g. as here (for the sake of simplicity without exception handling):

    std::vector<unsigned char> signData(EVP_PKEY* privateKey, const std::vector<unsigned char>& hashData) {
    EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new(privateKey, NULL);
    EVP_PKEY_sign_init(ctx);
    EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PSS_PADDING);
    EVP_PKEY_CTX_set_signature_md(ctx, EVP_sha256());
    //EVP_PKEY_CTX_set_rsa_mgf1_md(ctx, EVP_sha256()); // for this use case optional
    //EVP_PKEY_CTX_set_rsa_pss_saltlen(ctx, RSA_PSS_SALTLEN_DIGEST); // for this use case optional
    size_t sigLen = 0;
    EVP_PKEY_sign(ctx, NULL, &sigLen, hashData.data(), hashData.size());
    std::vector<unsigned char> signature(sigLen);
    EVP_PKEY_sign(ctx, signature.data(), &sigLen, hashData.data(), hashData.size());
    EVP_PKEY_CTX_free(ctx);
    return signature;
}

    For compatibility with the Python code, please note the following in the above C code: