I am working some more with an RC4 C++ program to be compatible with the command line arguments. I was wondering if somebody can point me in the direction of some decent documentation of how the command line openssl rc4 uses the salts when encrypting vs decrypting, so I can incorporate some compatibility into my program. Thanks to some help from someone on here a few days ago, my program is compatible with the non-salted version, once the EVP_bytestokey function is incorporated. I looked into the EVP_bytestokey function, which openssl uses, and the documentation shows that it can take a salt parameter:
"The salt parameter is used as a salt in the derivation: it should point to an 8 byte buffer or NULL if no salt is used."
I created an array of unsigned char's with the salt that the CLI gave me, and stored them in the array as their ASCII equivalents to pass in (to EVP_bytestokey) as the SALT parameter. Then I compared the file sizes and it shows that the encrypted/salted version of the file is 16 bytes larger than the original file. It appears the CLI version prepends "salted__" to the file, but that only accounts for 8 of the 16 bytes. Does anyone have any idea where the extra 8 bytes would be coming from? As far as I understand, the salt is prepended to the passphrase before passing into the keystream generator for RC4_setkey so I don't understand why the file size should be changed, outside of "salted__".
Also, do you think I am going in the right direction with the SALT array, storing the hex values as ASCII equivalents?
I have the documentation for the C functions being used here, but I can't seem to find anything specific about the mechanics of the CLI version, so I would appreciate any help I can get here.
So for anyone who is curious, I figured it out. OpenSSL RC4 with salt option prepends 16 bytes to the beginning on the file after encryption. The first 8 bytes are: "salted__" and the second 8 bytes are the actual salt, shown as unsigned char's. Lseek can be used to skip bytes 1-8, then bytes 9-16 can be loaded into an unsigned char array (Has to be size 8 bytes). This array should then be cast to (const unsigned char*) while being passed in as the salt to EVP_bytestokey(), which is the 3rd parameter.
As for a user input option for the salt, I set it up to take in two digit hex inputs into an int variable, which I then pass into the unsigned char salt array.