Openssl C API "EVP_BytesToKey()" returns wrong key and iv when used

Using the OpenSSL C API , I want to decrypt a file using a password, in order to do that I need to manually derive the key and iv from the given password using EVP_BytesToKey() , but the function gives me the wrong key and iv.

I encrypt the file using the command line tool like so :

openssl enc -nosalt -p -in sample.txt -out sample.txt.enc -e -aes256 -k PASSWORD

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
key=0BE64AE89DDD24E225434DE95D501711339BAEEE18F009BA9B4369AF27D30D60
iv =44A783DFFE1463B81E65064750797FA4

Here is my code (most of it is from examples I've seen of stack overflow) :

#include <string>
#include <fstream>

#include <openssl/ssl.h>

#include <iostream>
#include <vector>
#include <locale>
#include <windows.h>


int main(){
 // Initializing OpenSLL
     const EVP_CIPHER *cipher;
    const EVP_MD *dgst = NULL;
    unsigned char key[EVP_MAX_KEY_LENGTH], iv[EVP_MAX_IV_LENGTH];

    // Setting password

    const char *password = "PASSWORD";
    const unsigned char *salt = NULL;
    int i;

    OpenSSL_add_all_algorithms();


    // Getting key and iv

    cipher = EVP_get_cipherbyname("aes-256-cbc");
    if(!cipher) { fprintf(stderr, "no such cipher\n"); return 1; }

    dgst=EVP_get_digestbyname("md5");
    if(!dgst) { fprintf(stderr, "no such digest\n"); return 1; }

    if(!EVP_BytesToKey(cipher, dgst, salt,
        (unsigned char *) password,
        strlen(password), 1, key, iv))
    {
        fprintf(stderr, "EVP_BytesToKey failed\n");
        return 1;
    }


    // Output key and iv to the screen

    printf("Key: "); for(i=0; i< 32; ++i) { printf("%02x", key[i]); } printf("\n");
    printf("IV: "); for(i=0; i< 16; ++i) { printf("%02x", iv[i]); } printf("\n");
}

When debugging, the program outputs the key and iv to the screen , they contains random bytes?!?

Somehow when printing the key and iv to the screen with "%02x" it shows valid hex values.

Here is what my program outputs :

Key: 319f4d26e3c536b5dd871bb2c52e3178dcbc5de3a413ea043012cf3506b6956e
IV: c23b1986151650adf58ba93c7a10f73f

The Output is different from the original key and iv. I've seen other people do this and it worked for them.

Thank you

Solution:

(read to the end for an improvement)

Since openSSL 1.1, specifying no digest outputs the SHA256 digest instead of the MD5. If you run this command (note the -md switch):

openssl enc -nosalt -p -in sample.txt -out sample.txt.enc -e -aes256 -k PASSWORD -md md5

You get the following output:

iv =C23B1986151650ADF58BA93C7A10F73F

Which is the same as your C program.

Conversely, if you switch your C source to sha256 digest:

dgst=EVP_get_digestbyname("sha256");

You obtain the same output as from the command line.

Improvement

As you might have noticed, the openssl gives you a warning *** WARNING : deprecated key derivation used. Looking into the openssl documentation we have an explanation for such warning:

Newer applications should use a more modern algorithm such as PBKDF2 as defined in PKCS#5v2.1 and provided by PKCS5_PBKDF2_HMAC.

We can then obtain the key directly with this algorithm both with openssl executable using the switch iter 1 which uses -pbkdf2 Use password-based key derivation function 2 automatically:

openssl enc -nosalt -p -in a.out -out a.out.enc -e -aes256 -k PASSWORD -md sha256 -iter 1

Getting

key=75753C1B22D09354380EB2DA277B994738CA4C5AC09191E6285E9F99E04AA7A0
iv =7AE8C8B15B535D4197B5EC8E59F65809

And modifying our C program to get the key with the same more secure algorithm:

#include "string.h"

#include "openssl/ssl.h"

int main(){
 // Initializing OpenSLL
     const EVP_CIPHER *cipher;
    const EVP_MD *dgst = NULL;
    unsigned char keyivpair[EVP_MAX_KEY_LENGTH+EVP_MAX_IV_LENGTH];
    unsigned char key[EVP_MAX_KEY_LENGTH], iv[EVP_MAX_IV_LENGTH];

    // Setting password

    const char *password = "PASSWORD";
    const unsigned char *salt = NULL;
    int i;

    OpenSSL_add_all_algorithms();


    // Getting key and iv

    cipher = EVP_get_cipherbyname("aes-256-cbc");
    if(!cipher) { fprintf(stderr, "no such cipher\n"); return 1; }

    dgst=EVP_get_digestbyname("sha256");
    if(!dgst) { fprintf(stderr, "no such digest\n"); return 1; }


    int iklen = EVP_CIPHER_key_length(cipher);
    int ivlen = EVP_CIPHER_iv_length(cipher);

    if( !PKCS5_PBKDF2_HMAC ((unsigned char *) password, 
                        strlen(password), 
                        salt, 
                        0, 
                        1, 
                        dgst, //sha256
                        iklen + ivlen,
                        keyivpair) )
    {
        fprintf(stderr, "PKCS5_PBKDF2_HMAC failed\n");
        return 1;       
    }
    
    memcpy(key, keyivpair, iklen);
    memcpy(iv, keyivpair + iklen, ivlen);

    // Output key and iv to the screen

    printf("Key: "); for(i=0; i< 32; ++i) { printf("%02x", key[i]); } printf("\n");
    printf("IV: "); for(i=0; i< 16; ++i) { printf("%02x", iv[i]); } printf("\n");

Which outputs:

Key: 75753c1b22d09354380eb2da277b994738ca4c5ac09191e6285e9f99e04aa7a0
IV: 7ae8c8b15b535d4197b5ec8e59f65809