How can I encrypte( aes-256-cbc) a file , in code (c++) and decrypt using openssl command line?

I have a small c++ code to encrypt a file using EVP_Cipher_ APIs.I have a requirement to decrypte the encrypted file using Openssl (command line). When tried to decrypt, it always ends up in Bad decrypt.

Note I can decrypt using EVP_Cipher apis.

openssl command I used

openssl enc -aes-256-cbs -d -K ae0479555eca5b5228a0ee00ecef15aab729f47ee881cf8b1d3ff18561a47290 -iv 3fa6d97f4807e145b37451fc344e58ca -in test.enc -out test.out -nosalt

the EVP API used

EVP_CIPHERInit_ex(ctx, EVP_aes_256_cbs(), null, key, iv, 1);
EVP_CipherUpdate();
EVP_CipherFinal_ex()
EVP_CIPHER_CTX_cleanip();

I can decrypt using same API, by putting last arugument to 0 in EVP_CIPHERInit_ex.

Also I kept 32 bytes key and iv value same as I would use with openssl cmd line.

Not sure whats going wrong? ANy pointer would help. Thanks!!

PS: The c++ code used:

using namespace std;

void encDecFile(const char * input_filepath, bool encrypt) {
  const unsigned char * key = (const unsigned char * )
  "ae0479555eca5b5228a0ee00ecef15aab729f47ee881cf8b1d3ff18561a47290";
  const unsigned char * iv = (const unsigned char * )
  "3fa6d97f4807e145b37451fc344e58ca";

  int intent = encrypt ? 1 : 0;
  std::string output_filepath = input_filepath;

  if (encrypt) {
    const char * ext = ".enc";
    output_filepath = output_filepath + std::string(".enc");
  } else {
    auto pos = output_filepath.find_last_of(".");
    output_filepath = output_filepath.substr(0, pos) + std::string(".out");
  }

  //const char *output_filepath = strncat(input_filepath, ext, sizeof(ext));
  ifstream input_file(input_filepath, std::ios::binary);
  ofstream output_file(output_filepath.c_str(), std::ios::binary);
  int out_len = 0;

  printf("encDecFile - Entry \n");

  if (!(input_file.is_open() && output_file.is_open())) {
    const char * whichFile = input_file.is_open() ? output_filepath.c_str() : input_filepath;
    printf("encDecFile Failed to open %s \n", whichFile);
    return;
  }

  input_file.seekg(0, std::ios::end);
  long long file_size = input_file.tellg();
  input_file.seekg(0, std::ios::beg);
  printf("encDecFile - Input file %s is of Size %lld is opened! \n", input_filepath, file_size);


  EVP_CIPHER_CTX * ctx;
  ctx = EVP_CIPHER_CTX_new();

  EVP_CipherInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv, 1);
  EVP_CIPHER_CTX_set_padding(ctx, 1);
  unsigned int blocksize = EVP_CIPHER_block_size(EVP_aes_256_cbc());

  char * in_buf = new char[2048];
  char * out_buf = new char[2048 + blocksize];

  int count = 0;
  while (!input_file.eof()) {
    input_file.read(in_buf, (1 << 20));
    std::streamsize dataSize = input_file.gcount();
    printf("encDecFile - Reading %d chunk of data of size %d \n", ++count, (int) dataSize);
    if (1 == EVP_CipherUpdate(ctx, (unsigned char * ) out_buf, & out_len, (const unsigned char * ) in_buf, (int) dataSize)) {
      output_file.write(out_buf, out_len);
      printf("encDecFile - Written %d bytes of encrypted data \n", out_len);
    } else {
      printf("Encryption failed after for %d Chunk! \n", count);
      break;
    }
  }
  if (1 == EVP_CipherFinal_ex(ctx, (unsigned char * ) out_buf, & out_len)) {
    output_file.write(out_buf, out_len);
    printf("encDecFile - Written %d bytes of final encrypted data \n", out_len);
  } else {
    printf("Encryption failed after to update Final \n");
  }

  input_file.close();
  output_file.close();

  delete[] out_buf;
  EVP_CIPHER_CTX_cleanup(ctx);
  EVP_CIPHER_CTX_free(ctx);

  printf("encDecFile - Done!! \n");

  return;
}

Encrypt decrypt using EVP_Cipher APi worked fine.

Solution

It's the hex

-- which is timely in the US, where October is a big advertising/merchandising campaign building up to Halloween on Oct. 31 which is (now) about magic, witchcraft, and the supernatural, or at least the popular conception(s) of them :-}

Commandline openssl ... -K ... -iv ... treats the strings as hex and converts them to bytes: 0xae 0x04 0x79 ... 0x90 and 0x3f 0xa6 ... 0xca . But your C++ code instead uses the character codes of the first half of the key string -- 0x6a 0x66 0x30 0x34 ... 0x61 0x61 -- and similarly the first half of the iv -- 0x33 0x66 0x61 0x36 ... 0x34 0x35. These are radically different and thus don't interoperate.

You can either have the compiler take the hex and compile to binary:

unsigned char key[] = {0xae,0x04,0x79, ... 0x90};
unsigned char iv[] = {0x3f,0xa6, ... 0xca};

or write code to convert the hex-character form to binary such as:

char * key_hex = "ae0479555eca5b5228a0ee00ecef15aab729f47ee881cf8b1d3ff18561a47290";
unsigned char key_bin[32];
for(int i = 0; i < sizeof key_bin; i++ ) sscanf(&key_hex[i*2], "%2hhx", &key_bin[i]);
// and similarly for iv, and use the _bin versions when calling EVP