I just can't seem to decrypt a hex buffer, which I'm pretty sure is encrypted with RC4 and which I'm pretty sure I know the key. Being a beginner in cryptography, I just want to make sure I'm actually doing everything right before starting to think that my assumptions are wrong.
const crypto = require('crypto');
const buffer = Buffer.from('471b...', 'hex');
const decipher = crypto.createDecipheriv('rc4', 'MyKey', '');
let decrypted = '';
decrypted += decipher.update(buffer, 'hex', 'utf8');
decrypted += decipher.final('utf8');
console.log(decrypted) // outputs stuff like "�Y6�k�"
Is my hex buffer really encrypted in RC4 and/or is my key right?
We cannot tell. The algorithm or key is probably not correct unless the input message was binary rather than text, because it doesn't look like any of the known character encodings.
Ciphertext is indistinguishable from random which makes detecting a modern cipher very hard (doubly so because you left out most of the ciphertext). RC4 can be distinguished however, but you need an intricate attack to distinguish it from random noise; that would presumably also identify the cipher, even without knowing the key.
Furthermore, RC4 can be initialized with almost any kind of key size. Smaller key sizes may be relatively easy to brute force - bigger key sizes might take forever though (quite literally, surviving beyond the heath death of the universe).
So answers in short:
By the way: print out the plaintext in hexadecimals in case you want to check if it makes sense or not. ASCII is easy to distinguish that way, but other schemes are also likely to show a pattern in binary, which you cannot see if you just get a diamond with a question mark in it (or any other replacement character, though some fonts / terminals actually display the hex value within the font, which is nice).