Inconsistent encryption and decryption with OpenSSL RC4 in C++

First off, I understand that RC4 is not the safest encryption method and that it is outdated, this is just for a school project. Just thought I put it out there since people may ask.

I am working on using RC4 from OpenSSL to make a simple encryption and decryption program in C++. I noticed that the encryption and decryption is inconsistent. Here is what I have so far:

#include <fcntl.h>
#include <openssl/evp.h>
#include <openssl/rc4.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
int inputFile = open(argv[1], O_RDONLY);
if (inputFile < 0) {
    printf("Error opening file\n");
    return 1;
}

unsigned char *keygen = reinterpret_cast<unsigned char*>(argv[2]);
RC4_KEY key;

size_t size = lseek(inputFile, 0, SEEK_END);
lseek(inputFile, 0, SEEK_SET);

unsigned char *fileIn = (unsigned char*) calloc(size, 1);

if (pread(inputFile, fileIn, size, 0) == -1) {
    perror("Error opening read\n");
    return 1;
}

unsigned char *fileOut = (unsigned char*) calloc(size, 1);

unsigned char *actualKey;
EVP_BytesToKey(EVP_rc4(), EVP_sha256(), NULL, keygen, sizeof(keygen), 1, actualKey, NULL);

RC4_set_key(&key, sizeof(actualKey), actualKey);
RC4(&key, size, fileIn, fileOut);

int outputFile = open(argv[3], O_WRONLY | O_TRUNC | O_CREAT, 0644);
if (outputFile < 0) {
    perror("Error opening output file");
    return 1;
}
if (pwrite(outputFile, fileOut, size, 0) == -1) {
    perror("error writing file");
    return 1;
}

close(inputFile);
close(outputFile);

free(fileIn);
free(fileOut);

return 0;
}

The syntax for running this in Ubuntu is:

./myRC4 test.txt pass123 testEnc.txt

MOST of the time this works fine, and encrypts and decrypts the file. However occasionally I get a Segmentation fault. If I do, I run the same exact command again and it encrypts or decrypts fine, at least for .txt files.

When I test on .jpg files, or any larger file, the issue seems to be more common and inconsistent. I notice that sometimes the images appear to have been decrypted (no segmentation fault) but in reality it has not, which I test by doing a diff between the original and the decrypted file.

Any ideas as to why I get these inconsistencies? Does it have to do with how I allocate memory for fileOut and fileIn?

Thank you in advance

actualKey needs to be pointing to a buffer of appropriate size before you pass it to EVP_BytesToKey. As it is you are passing in an uninitialised pointer which would explain your inconsistent results.

The documentation for EVP_BytesToKey has this to say:

If data is NULL, then EVP_BytesToKey() returns the number of bytes needed to store the derived key.

So you can call EVP_BytesToKey once with the data parameter set to NULL to determine the length of actualKey, then allocate a suitable buffer and call it again with actualKey pointing to that buffer.

As others have noted, passing sizeof(keygen) to EVP_BytesToKey is also incorrect. You probably meant strlen (argv [2]).

Likewise, passing sizeof(actualKey) to RC4_set_key is also an error. Instead, you should pass the value returned by EVP_BytesToKey.