Understanding the ZMODEM protocol

I need to include basic file-sending and file-receiving routines in my program, and it needs to be through the ZMODEM protocol. The problem is that I'm having trouble understanding the spec.

For reference, here is the specification.

The spec doesn't define the various constants, so here's a header file from Google.

It seems to me like there are a lot of important things left undefined in that document:

• It constantly refers to ZDLE-encoding, but what is it? When exactly do I use it, and when don't I use it?
• After a ZFILE data frame, the file's metadata (filename, modify date, size, etc.) are transferred. This is followed by a ZCRCW block and then a block whose type is undefined according to the spec. The ZCRCW block allegedly contains a 16-bit CRC, but the spec doesn't define on what data the CRC is computed.
• It doesn't define the CRC polynomial it uses. I found out by chance that the CRC32 poly is the standard CRC32, but I've had no such luck with the CRC16 poly. Nevermind, I found it through trial and error. The CRC16 poly is 0x1021.

I've looked around for reference code, but all I can find are unreadable, undocumented C files from the early 90s. I've also found this set of documents from the MSDN, but it's painfully vague and contradictory to tests that I've run: http://msdn.microsoft.com/en-us/library/ms817878.aspx (you may need to view that through Google's cache)

To illustrate my difficulties, here is a simple example. I've created a plaintext file on the server containing "Hello world!", and it's called helloworld.txt.

I initiate the transfer from the server with the following command:

sx --zmodem helloworld.txt

This prompts the server to send the following ZRQINIT frame:

2A 2A 18 42 30 30 30 30 30 30 30 30 30 30 30 30   **.B000000000000
30 30 0D 8A 11                                    00.Š.

Three issues with this:

• Are the padding bytes (0x2A) arbitrary? Why are there two here, but in other instances there's only one, and sometimes none?
• The spec doesn't mention the [CR] [LF] [XON] at the end, but the MSDN article does. Why is it there?
• Why does the [LF] have bit 0x80 set?

After this, the client needs to send a ZRINIT frame. I got this from the MSDN article:

2A 2A 18 42 30 31 30 30 30 30 30 30 32 33 62 65   **.B0100000023be
35 30 0D 8A                                       50.Š

In addition to the [LF] 0x80 flag issue, I have two more issues:

• Why isn't [XON] included this time?
• Is the CRC calculated on the binary data or the ASCII hex data? If it's on the binary data I get 0x197C, and if it's on the ASCII hex data I get 0xF775; neither of these are what's actually in the frame (0xBE50). (Solved; it follows whichever mode you're using. If you're in BIN or BIN32 mode, it's the CRC of the binary data. If you're in ASCII hex mode, it's the CRC of what's represented by the ASCII hex characters.)

The server responds with a ZFILE frame:

2A 18 43 04 00 00 00 00 DD 51 A2 33               *.C.....ÝQ¢3

OK. This one makes sense. If I calculate the CRC32 of [04 00 00 00 00], I indeed get 0x33A251DD. But now we don't have ANY [CR] [LF] [XON] at the end. Why is this?

Immediately after this frame, the server also sends the file's metadata:

68 65 6C 6C 6F 77 6F 72 6C 64 2E 74 78 74 00 31   helloworld.txt.1
33 20 32 34 30 20 31 30 30 36 34 34 20 30 20 31   3 240 100644 0 1
20 31 33 00 18 6B 18 50 D3 0F F1 11                13..k.PÓ.ñ.

This doesn't even have a header, it just jumps straight to the data. OK, I can live with that. However:

• We have our first mysterious ZCRCW frame: [18 6B]. How long is this frame? Where is the CRC data, and is it CRC16 or CRC32? It's not defined anywhere in the spec.
• The MSDN article specifies that the [18 6B] should be followed by [00], but it isn't.
• Then we have a frame with an undefined type: [18 50 D3 0F F1 11]. Is this a separate frame or is it part of ZCRCW?

The client needs to respond with a ZRPOS frame, again taken from the MSDN article:

2A 2A 18 42 30 39 30 30 30 30 30 30 30 30 61 38   **.B0900000000a8
37 63 0D 8A                                       7c.Š

Same issues as with the ZRINIT frame: the CRC is wrong, the [LF] has bit 0x80 set, and there's no [XON].

The server responds with a ZDATA frame:

2A 18 43 0A 00 00 00 00 BC EF 92 8C               *.C.....¼ï’Œ

Same issues as ZFILE: the CRC is all fine, but where's the [CR] [LF] [XON]?

After this, the server sends the file's payload. Since this is a short example, it fits in one block (max size is 1024):

48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0A            Hello world!.

From what the article seems to mention, payloads are escaped with [ZDLE]. So how do I transmit a payload byte that happens to match the value of [ZDLE]? Are there any other values like this?

The server ends with these frames:

18 68 05 DE 02 18 D0                              .h.Þ..Ð
2A 18 43 0B 0D 00 00 00 D1 1E 98 43               *.C.....Ñ.˜C

I'm completely lost on the first one. The second makes as much sense as the ZRINIT and ZDATA frames.

My buddy wonders if you are implementing a time machine.

I don't know that I can answer all of your questions -- I've never actually had to implement zmodem myself -- but here are few answers:

From what the article seems to mention, payloads are escaped with [ZDLE]. So how do I transmit a payload byte that happens to match the value of [ZDLE]? Are there any other values like this?

This is explicitly addressed in the document you linked to at the beginning of your questions, which says:

The ZDLE character is special.  ZDLE represents a control sequence
of some sort.  If a ZDLE character appears in binary data, it is
prefixed with ZDLE, then sent   as ZDLEE.

It constantly refers to ZDLE-encoding, but what is it? When exactly do I use it, and when don't I use it?

In the Old Days, certain "control characters" were used to control the communication channel (hence the name). For example, sending XON/XOFF characters might pause the transmission. ZDLE is used to escape characters that may be problematic. According to the spec, these are the characters that are escaped by default:

ZMODEM software escapes ZDLE, 020, 0220, 021, 0221, 023, and 0223.
If preceded by 0100 or 0300 (@), 015 and 0215 are also escaped to
protect the Telenet command escape CR-@-CR.  The receiver ignores
021, 0221, 023, and 0223 characters in the data stream.

I've looked around for reference code, but all I can find are unreadable, undocumented C files from the early 90s.

Does this include the code for the lrzsz package? This is still widely available on most Linux distributions (and surprisingly handy for transmitting files over an established ssh connection).

There are a number of other implementations out there, including several in software listed on freecode, including qodem, syncterm, MBSE, and others. I believe the syncterm implementation is written as library that may be reasonable easy to use from your own code (but I'm not certain).

You may find additional code if you poke around older collections of MS-DOS software.