OpenSSL 3 Diffie-Hellman Key Exchange C++

Before OpenSSL3 this was simple.

DH* dh = DH_new();

/* Parameters */
dh->p = BN_bin2bn(bin_p, bin_p_size, NULL);
dh->g = BN_bin2bn(bin_g, bin_g_size, NULL);

/* Private key generation */
BN_hex2bn(&dh->priv_key, hex_priv_key);

/* Public key generation */
DH_generate_key(dh);

/* Derive */
int shared_key_size = DH_compute_key(shared_key, peer_pub_key, dh);

I am trying to make keys in the new version of OpenSSL but it didnt work because EVP_PKEY_generate fails with error:03000097:digital envelope routines::operation not initialized

OSSL_PARAM_BLD* param_build = OSSL_PARAM_BLD_new();
OSSL_PARAM_BLD_push_BN(param_build, OSSL_PKEY_PARAM_FFC_P, p);
OSSL_PARAM_BLD_push_BN(param_build, OSSL_PKEY_PARAM_FFC_G, g);

OSSL_PARAM* params = OSSL_PARAM_BLD_to_param(param_build};

/* DH_new() */
EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new_from_name(nullptr, "DH", nullptr);
EVP_PKEY_keygen_init(ctx);

/* DH_generate_key */
EVP_PKEY* dh_key_pair = NULL;
EVP_PKEY_generate(ctx, &dh_key_pair, EVP_PKEY_KEY_PARAMETERS, params);

There is lack of information on the Internet. I dont know what to do.

Solution

You have to create an EVP_PKEY with domain parameters first if you use custom prime and generator.

// Create the OSSL_PARAM_BLD.
OSSL_PARAM_BLD* paramBuild = OSSL_PARAM_BLD_new();
if (!paramBuild) {
  // report the error
}
// Set the prime and generator.
if (!OSSL_PARAM_BLD_push_BN(paramBuild, OSSL_PKEY_PARAM_FFC_P, prime) ||
    !OSSL_PARAM_BLD_push_BN(paramBuild, OSSL_PKEY_PARAM_FFC_G, generator)) {
  // report the error
}
// Convert to OSSL_PARAM.
OSSL_PARAM* param = OSSL_PARAM_BLD_to_param(paramBuild);
if (!param) {
  // report the error
}
// Create the context. The name is DHX not DH!!!
EVP_PKEY_CTX* domainParamKeyCtx = EVP_PKEY_CTX_new_from_name(nullptr, "DHX", nullptr);
if (!domainParamKeyCtx) {
  // report the error
}
// Initialize the context.
if (EVP_PKEY_fromdata_init(domainParamKeyCtx) <= 0) {
  // report the error
}
// Create the domain parameter key.
EVP_PKEY* domainParamKey = nullptr;
if (EVP_PKEY_fromdata(domainParamKeyCtx, &domainParamKey,
    EVP_PKEY_KEY_PARAMETERS, param) <= 0) {
  // report the error
}

The domain parameter domainParamKey is ready! Now, use it to create a key pair.

EVP_PKEY_CTX* keyGenerationCtx = EVP_PKEY_CTX_new_from_pkey(nullptr, domainParamKey, nullptr);
if (!keyGenCtx) {
  // report the error
}
if (EVP_PKEY_keygen_init(keyGenerationCtx) <= 0) {
  // report the error
}
EVP_PKEY* keyPair = nullptr;
if (EVP_PKEY_generate(keyGenerationCtx, &keyPair) <= 0) {
  // report the error
}

The key pair keyPair is ready!

See this project on GitHub: https://github.com/eugen15/diffie-hellman-cpp. It shows how to generate key pairs, set and get Diffie-Hellman parameters. Plus, it has some custom Diffie-Hellman implementations.

See the following files for an OpenSSL 3 example:

diffie-hellman-openssl.h
diffie-hellman-openssl.cpp

The following files is a LibreSSL example which is basically OpenSSL before version 3. This is to be sure you have backward compatibility.

diffie-hellman-libressl-dh.h
diffie-hellman-libressl-dh.cpp

More things to consider

FIRST

If you used a custom prime before OpenSSL 3, then it is better to move to a predefined safe prime. You can see an example here: https://www.openssl.org/docs/manmaster/man7/EVP_PKEY-DH.html

int priv_len = 2 * 112;
OSSL_PARAM params[3];
EVP_PKEY *pkey = NULL;
EVP_PKEY_CTX *pctx = EVP_PKEY_CTX_new_from_name(NULL, "DH", NULL);

params[0] = OSSL_PARAM_construct_utf8_string("group", "ffdhe2048", 0);
/* "priv_len" is optional */
params[1] = OSSL_PARAM_construct_int("priv_len", &priv_len);
params[2] = OSSL_PARAM_construct_end();

EVP_PKEY_keygen_init(pctx);
EVP_PKEY_CTX_set_params(pctx, params);
EVP_PKEY_generate(pctx, &pkey);
...
EVP_PKEY_free(pkey);
EVP_PKEY_CTX_free(pctx);

Alice executes the above code (taken from the OpenSSL manual). Then you get the prime and generator:

EVP_PKEY_get_bn_param(pkey, OSSL_PKEY_PARAM_FFC_P, &prime);
EVP_PKEY_get_bn_param(pkey, OSSL_PKEY_PARAM_FFC_G, &generator);

Then send it to Bob and use my code above to create the key pair.

SECOND

Use Elliptic-Curve Diffie-Hellman (ECDH) key exchange for new projects. It is much faster. Leave your DH as a fallback. See examples here.