RC4 implementation doesn't match openssl output

My goal is to implement the RC4 stream cipher in C/C++, and make sure it produces the same output as when using the openssl command. Following the pseudocode on wikipedia, this implementation appears to work, in that it can encrypt and decrypt content. However, the encrypted output doesn't match the output of the equivalent openssl command, I'd like to understand why.

/*
    Reference: https://en.wikipedia.org/wiki/RC4
*/

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

typedef u_int8_t byte;
typedef struct
{
    byte i;
    byte j;
    byte S[256];
} Rc4State;

static void swap(byte *a, byte *b)
{
    byte temp = *a;
    *a = *b;
    *b = temp;
}

/*
    Set initial permutation.
    Initialize i and j counters for stream generation.
*/
void rc4InitState(Rc4State *state, const byte K[256], u_int8_t klen)
{
    byte *S = state->S;

    for (int i = 0; i < 256; i++)
    {
        S[i] = i;
    }

    int j = 0;
    for (int i = 0; i < 256; i++)
    {
        j = (j + S[i] + K[i % klen]) % 256;
        swap(&S[i], &S[j]);
    }

    state->i = 0;
    state->j = 0;
}

void rc4Crypt(Rc4State *state, byte buffer[], size_t len)
{
    byte *S = state->S;

    for (size_t k = 0; k < len; k++)
    {
        byte i = ++(state->i);
        byte j = (state->j += S[i]);

        swap(&S[i], &S[j]);
        byte t = S[i] + S[j];

        buffer[k] ^= S[t];
    }
}

int main(int argc, const char *argv[])
{
    if (argc < 2)
    {
        printf("usage: %s secret [-nosalt]\n", argv[0]);
        exit(1);
    }

    const char *key = argv[1];
    int klen = strlen(key);

    if (!(1 <= klen && klen < 256))
    {
        printf("secret length must be between 1 and 255 chars, got %d\n", klen);
        exit(1);
    }

    byte *K = new byte[256];
    for (int i = 0; i < klen; i++)
    {
        K[i] = key[i];
    }

    Rc4State *state = (Rc4State *)malloc(sizeof(Rc4State));
    rc4InitState(state, K, klen);

    byte *buffer = new byte[256];
    while (1)
    {
        size_t size = fread(buffer, sizeof(byte), 256, stdin);
        if (ferror(stdin))
            break;

        rc4Crypt(state, buffer, size);
        fwrite(buffer, sizeof(byte), size, stdout);

        if (feof(stdin))
            break;
    }
}

I compile it with:

g++ prog.cpp

Where g++ --version is:

$ g++ --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.29)
Target: x86_64-apple-darwin19.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I test it with:

path=path/to/some/file
./a.out foo < "$path" > /tmp/a
openssl rc4 -pass pass:foo -nosalt < "$path" > /tmp/b
cmp /tmp/a /tmp/b

Unfortunately the outputs don't match, I'd like to understand why.

In case it helps, consider this concrete example:

for i in {1..100}; do echo $i; done > /tmp/sample.txt

Head of hexdump using my implementation:

$ ./a.out foo < /tmp/sample.txt | hexdump -C | head
00000000  9a 28 df 23 fb 6b 6f 38  03 80 83 ff aa a4 6b 81  |.(.#.ko8......k.|
00000010  5d 28 e8 a7 bf b1 4d d0  4b 34 3c 65 7c 21 f9 26  |](....M.K4<e|!.&|
00000020  d0 aa 2f 75 e6 96 9c d3  df 64 54 3b 6e 5b cc 47  |../u.....dT;n[.G|
00000030  50 76 23 48 f6 88 f8 c7  88 47 f5 89 4f 3e 01 5b  |Pv#H.....G..O>.[|
00000040  e1 b4 f9 03 f3 56 48 9c  c2 a1 45 dc a1 ed da ce  |.....VH...E.....|
00000050  99 1e d2 ab 65 29 d8 8f  49 a3 bf 88 7c 49 d2 9a  |....e)..I...|I..|
00000060  78 f7 ed 04 ec 23 f4 8a  18 06 7d ec 74 90 12 60  |x....#....}.t..`|
00000070  94 f9 a5 9b f8 97 c4 9b  31 94 eb dd 32 66 5e 8a  |........1...2f^.|
00000080  03 4d c1 d1 75 b5 89 9b  19 1f 6f 55 39 59 97 78  |.M..u.....oU9Y.x|
00000090  c6 64 81 85 8e 9c b8 0f  ef 29 90 77 29 02 0e 93  |.d.......).w)...|

Head of hexdump using openssl (on OSX, version OpenSSL 1.0.1e 11 Feb 2013):

$ openssl rc4 -pass pass:foo -nosalt < /tmp/sample.txt | hexdump -C | head
00000000  de c0 1e 94 70 ef f2 55  45 69 f0 c9 71 94 30 32  |....p..UEi..q.02|
00000010  b4 6e fd d5 43 ef 4c 56  a0 58 00 8e f2 33 84 cd  |.n..C.LV.X...3..|
00000020  e2 d4 14 3b 78 7c 27 34  1a f2 2c e5 3a c2 9a 6e  |...;x|'4..,.:..n|
00000030  ab 20 e5 30 84 4f 17 b5  1a 2f 76 f6 b2 30 48 81  |. .0.O.../v..0H.|
00000040  39 70 50 21 f2 fc dc 0b  11 eb 0e e8 fa 0f ab 7c  |9pP!...........||
00000050  02 28 0a 0e 06 8e f6 44  2b 0d 67 c0 88 12 a7 74  |.(.....D+.g....t|
00000060  66 a3 18 b5 f8 ea d4 7b  05 84 cf 56 42 07 0c 8c  |f......{...VB...|
00000070  d9 8a c3 fc dc 30 9a ef  4c ca 00 b8 7d 15 32 ac  |.....0..L...}.2.|
00000080  7d 3e 54 19 d8 b6 a6 22  d1 14 a1 a4 12 b0 a5 aa  |}>T...."........|
00000090  63 e1 63 87 8b a1 58 88  6c 19 3d 32 09 07 0f bf  |c.c...X.l.=2....|

What am I missing?

Interestingly, the same openssl command on the same sample gives a different result on a Debian system (version OpenSSL 1.1.1c 28 May 2019):

$ openssl rc4 -pass pass:foo -nosalt < /tmp/sample.txt | hexdump -C | head
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
00000000  13 b9 ff 0f bf 7e e7 4b  6b 6f 28 86 10 a5 a0 cd  |.....~.Kko(.....|
00000010  86 6d 6b d5 58 8e c1 d0  65 3e cd ae b0 c5 64 5f  |.mk.X...e>....d_|
00000020  f8 fc 93 76 1e ce c2 d3  8e 1a e1 7d 78 12 ee 3a  |...v.......}x..:|
00000030  b1 a8 43 ec c4 fb 06 ed  f9 3a fd 8d c7 d9 18 4d  |..C......:.....M|
00000040  e5 a4 6a 9e 59 f4 a6 37  b3 0d 6f c8 cb a4 fb 6a  |..j.Y..7..o....j|
00000050  31 8a 3a 5e f3 df 41 6f  d2 3d 53 aa ee 6f cb 31  |1.:^..Ao.=S..o.1|
00000060  8b 43 e8 f5 45 91 46 4c  15 ab d1 0e 5d 6a 19 90  |.C..E.FL....]j..|
00000070  c7 fc f2 10 89 e3 bb 1d  33 d7 9c 42 70 31 bf 05  |........3..Bp1..|
00000080  e1 dc 91 47 92 a9 d9 da  de c3 a1 b3 20 5d c7 d5  |...G........ ]..|
00000090  d1 6f 8e 57 05 f6 6e 87  38 49 bc d8 90 29 9a 4d  |.o.W..n.8I...).M|

So perhaps the openssl implementation/version is important?

Solution

rc4Crypt from the posted code provides the same ciphertext as OpenSSL. Concerning OpenSSL the following has to be taken into account:

The key must be set with the -K option. In contrast, the -pass option passes a password from which the key is derived.
OpenSSL only supports certain key lengths: rc4 supports 16 bytes key and rc4-40 supports 5 bytes key. With openssl enc -ciphers the supported algorithms can be listed (from version 1.1.0 on).

Examples:

Plaintext:       test
RC4-Key:         tests
OpenSSL command: openssl rc4-40 -K 7465737473 -nosalt -p -in plaintext.txt -out ciphertext.txt
Result:          DD9B5CB9

Plaintext:       AnotherTest
RC4-Key:         My16BytesTestKey
OpenSSL command: openssl rc4 -K 4d7931364279746573546573744b6579 -nosalt -p -in plaintext.txt -out ciphertext.txt
Result:          425E42CC1FD9F0E066A227

More test vectors can be found here.

The RC4-key foo cannot be tested with OpenSSL because this key size isn't supported. However, it can be tested here. Again rc4Crypt returns the same ciphertext, e.g.:

Plaintext:       AThirdTest
RC4-Key:         foo
RC4-Key (hex):   666f6f
Result:          EA768540BA050F5745FE