Mostly of a historical interest and if I were to implement 8086 compatibility for assembler, what operands are considered valid for ESC instruction?
ESC opcode, source
From 8086 Programmer's manual I know, that opcode is an immediate in range 0 to 63 and source is a register or memory. But what registers can be encoded? Both reg8 and reg16 or only reg16? If source is memory, does operand size (mem8 or mem16) matter?
Basically, both of the above don't really matter from instruction encoding perspective (as, for example, both esc 0x01, ch and esc 0x01, bp would produce the same result), but maybe assemblers had forced restrictions
And, the last, but not the least, where can I find description for ESC opcodes?
The 8086 has an opcode space collectively designated ESC (escape to coprocessor). It occupies the range d8 to df. Each instruction in this instruction space is followed by a modr/m byte and depending on the mod-field, zero to two displacement bytes. When the 8086 encounters an ESC instruction with two register operands (i.e. mod = 11), it performs a nop. When the processor encounters an ESC instruction with a memory operand, a read cycle is performed from the address indicated by the memory operand and the result is discarded.
Using two special signal lines, a coprocessor can distinguish data fetches from instruction fetches, allowing it to decode the instruction stream in parallel with the 8086.
This mechanism is used by the 8087 to hook into the instruction stream: The three available bits in the opcode byte together with the three bits of the reg field in the modr/m byte form a six bit opcode. The r/m field of the modr/m byte is used to designate either a location on the FPU register stack (if mod = 11, indicating two register operands) or a memory operand. Some opcodes encode a variety of instructions depending on the content of r/m field. In all these cases, one instruction is encoded for memory operands while eight other instructions are encoded for each possible register operand.
The 8087 registers when the 8086 performs the dummy fetch immediately after fetching the instruction and remembers the address. In case of an instruction that loads from memory, it loads the additional words of the memory operand and performs its function. In case of a store, it ignores the result of the fetch and stores its values to the address indicated by the 8086.
The 8087 performs operation asynchronously to the 8086. However, no implicit synchronization exists. If an attempt is made to issue an ESC instruction while the coprocessor is busy, the instruction is silently ignored. To solve this problem, the 8087 asserts its BUSY pin (connected to the 8086's TEST pin) while performing operation. The programmer can issue a wait instruction (9b, wait for coprocessor ready) to wait until the 8087 has concluded operation and consequently released the WAIT line. This is typically done before every single 8087 instruction and many assemblers of the day would automatically insert WAIT prefixes. For high performance code, it was also common to instead manually calculate how long the 8087 was going to take for a certain instruction and to omit the wait when the previous floating point instruction was guaranteed to have finished.