I've read through section 20 of volume 3 of the Intel Software Developer's Manual, which explains the workings of virtual-8086 mode, including the use of the VIF and VIP flags. However, I'm still confused about some things.
The VIF flag is used as a surrogate for the IF flag, so that an 8086 emulator (which is presumably some user-space program) can receive maskable hardware interrupts even if the real-mode program it's emulating doesn't want to.
The VIP is used as a space to mark when an interrupt is received. According to the manual, if the processor receives a maskable interrupt, but the program being emulated doesn't want to receive them (so the VIF flag would be cleared), the emulator should set the VIP flag, wait until the emulated program sets the IF flag, and then handle the interrupt.
Why do we wait to handle the interrupt instead of handling it immediately? Are interrupts on a real-mode 8086 deferred rather than ignored?
Additional question—the processor generates a #GP when the VIP is set and the program enables interrupts (through STI etc.) But a #GP could be generated while the VIP flag is set in multiple ways—e.g. through an invalid memory access. The manual doesn't make any mention of checking what the cause of the #GP was, so should the program process the pending interrupts regardless of whether the IF was being set?
Edit: Okay, so I see part of my confusion—it's the kernel which needs to keep the IF set. In that case I should rather ask for #2—why do we wait to handle the interrupt instead of ignoring it entirely?
The reason for this is covered in section 20.3.2: "Existing 8086 programs commonly set and clear the IF flag in the EFLAGS register to enable and disable maskable hardware interrupts." When running such an application in virtual-8086 mode under a modern OS, the OS is not going to allow the 8086 program to disable interrupts in the system. Thus the IF flag needs to remain under control of the OS, and it must be virtualized on behalf of the virtual-8086 program. Use of the VIF and VIP flags "eliminat[es] the need for all IF related operations (such as PUSHF, POPF, CLI, and STI instructions) to trap to the virtual-8086 monitor."
The interrupt cannot be delivered to the virtual-8086 mode program if it has executed CLI to clear IF (which has the effect of clearing VIF).
That's a good question. It seems to me that the processor should set VIF before generating #GP; then the GP handler in the virtual-8086 monitor can easily see that VIP and VIF are both set. This is the described behavior for when VIF is set by a POPF or IRET; I don't know why STI behaves differently.