I am looking at a quite old DES library (d3des), which is still used today in VNC servers: d3des in UltraVNC. I am assuming that the code correctly implements the DES specification, as it's been in use for quite some time now. However, I cannot seem to get the same results from using this code as I get from using DES in Python, C# or Java. For the latter ones, I get the same, consistent results. However, these results don't match the results from d3des. So I wrote a small program to verify the d3des implementation according to test vectors I got from Validating the Correctness of Hardware Implementations of the NBS Data Encryption standard. The very first test vector is
Key: 01 01 01 01 01 01 01 01
Plain: 95 F8 A5 E5 DD 31 D9 00
Cipher: 80 00 00 00 00 00 00 00
My test program (C++) looks like this (I am omitting the printx() function as it is used only for printing data):
void sampledes(unsigned char* key, unsigned char* plain) {
unsigned char msg[8];
deskey(key, EN0);
des(plain, msg);
printx("key: ", key, 8);
printx("plain: ", plain, 8);
printx("msg: ", msg, 8);
}
int main()
{
sampledes(
new unsigned char[] { 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01 },
new unsigned char[] { 0x95, 0xf8, 0xa5, 0xe5, 0xdd, 0x31, 0xd9, 0x00 } );
}
The output is:
key: 1 1 1 1 1 1 1 1
plain: 95 f8 a5 e5 dd 31 d9 0
msg: 3e 5a 14 8d cd f e9 ad
Of course, I would expect the value written above as the msg value, which I am getting from a DES encryption in Java, for example.
I suspected some issue with byte ordering, so I tried some permutations of the plain text value, but to no avail (key is the same, plain value, cipher value):
01 01 01 01 01 01 01 01 95 f8 a5 e5 dd 31 d9 00 3e 5a 14 8d cd 0f e9 ad
01 01 01 01 01 01 01 01 dd 31 d9 00 95 f8 a5 e5 1f bd a0 58 5e 3d aa d0
01 01 01 01 01 01 01 01 f8 95 e5 a5 31 dd 00 d9 47 cd c4 cf fb 53 22 0b
01 01 01 01 01 01 01 01 00 d9 31 dd e5 a5 f8 95 c3 08 39 95 44 15 e9 3f
01 01 01 01 01 01 01 01 e5 a5 f8 95 00 d9 31 dd d5 a7 f3 87 f3 aa 33 02
I guess I am missing something obvious here as I cannot get the test vector right. Maybe some of you can spot an issue?
Addendum: I forgot to mention the weirdest thing of them all. In the d3des.c file mentioned above there are test vectors given. When I run the first one, I get:
01 23 45 67 89 ab cd ef 01 23 45 67 89 ab cd e7 df 78 18 5a 0d 55 c5 e5
However, as a result, c957 4425 6a5e d31d is expected. Note that this one here is the original notation, i.e. the data is given in pairs of bytes. This is what makes me think that originally they are not using byte arrays as the source of data but short integers or something like this. However, shuffling around the bytes does not give the expected result either:
23 01 67 45 ab 89 ef cd 23 01 67 45 ab 89 e7 cd 31 47 d6 be 46 d4 e6 50
Please note that I'm not trying to apply the outdated DES algorithm to anything new. Still, DES is used in the VNC protocol for password-based authentication (and it is recommended not to run the protocol over unencrypted lines). If I was trying to implement a VNC client to interact with an existing server, I needed to use this algorithm.
Edit: Okay, it seems this is a lost cause. In the d3des.c version of VNC, it says in the comments:
* This is D3DES (V5.09) by Richard Outerbridge with the double and
* triple-length support removed for use in VNC. Also the bytebit[] array
* has been reversed so that the most significant bit in each byte of the
* key is ignored, not the least significant.
*
* These changes are
* Copyright (C) 1999 AT&T Laboratories Cambridge. All Rights Reserved.
So they made some modifications to the original code, and the given test vectors were not updated. This also explains why I'm getting different results from other d3des.c versions found on Github.
Conclusion: As described in my comment below, I solved the issue by using the C code from the UltraVNC server, wrapping it in a separate DLL, and calling this DLL from my (.NET) code. @Topaco gives a great answer about the underlying issue. With this in mind, it would also be possible to use the native .NET implementation of DES (with some modification of the password).
According to this post, the bits of each key byte are to be mirrored. I have checked this with 5 of your test vectors and the code returns in each case the expected result when reversing the key bits.
Example code:
// from https://stackoverflow.com/a/2602885/9014097
unsigned char reverse(unsigned char b) {
b = (b & 0xF0) >> 4 | (b & 0x0F) << 4;
b = (b & 0xCC) >> 2 | (b & 0x33) << 2;
b = (b & 0xAA) >> 1 | (b & 0x55) << 1;
return b;
}
...
unsigned char key[] = { 0x7C, 0xA1, 0x10, 0x45, 0x4A, 0x1A, 0x6E, 0x57 };
unsigned char plainText[] = { 0x01, 0xA1, 0xD6, 0xD0, 0x39, 0x77, 0x67, 0x42 }; // 1st testvector on page 33, result: 690F5B0D9A26939B
unsigned char mkey[8];
unsigned char cipherText[8];
for (int i = 0; i < 8; i++) {
mkey[i] = reverse(key[i]);
}
deskey(mkey, EN0);
des(plainText, cipherText);
for (int i = 0; i < 8; ++i) {
printf("%02X", cipherText[i]);
} // 690F5B0D9A26939B
printf("\n");
...
Regarding the comment you found: Presumably this applies to 3DES (and the various keying options). However, DES is apparently still possible even with the changes (as described above).