I'm studying virtualization from the Tanenbaum's book ("Modern Operating Systems"). I have clearly in my mind all the basic concept of the virtualization, but i can't understand better how VMWare Workstation works.
First, the VMWare Workstation has two components:
The VMM uses before each (?) execution a "decision algorithm" to establish if can execute it using the "Direct Execution" (trap-and-emulate) or the "Binary Translation".
Tanenbaum said that a sensible-instruction generates a "trap" only in several cases (in this case the VMM can use the Direct Execution, improving the performance).
What isn't clear for me, is why on the x86 platform a sensible instruction, isn't sensible all the time (on the x86 platform), and in how circumstances that is true?
I haven't read the Tanenbaum's book, this is my interpretation of the author words.
The 18 sensitive instructions, according to Wikipedia, that cannot be run directly are:
sgdt(Read the GDT of the host, not of the guest)
sidt(As above, but for the IDT)
sldt(As above, but for the LDT)
smsw(Read the control register 0 of the host, not the guest one)
pushf(Read the flags, particularly system flags, of the host, not of the guest)
popf(As above, but write, only some, of the flags)
lar(Read the access right from the descriptors tables of the host, not of the guest)
lsl(Read the segment limit from the descriptors tables of the host, not of the guest)
verr,verw(Check for read/write access using the host descriptors tables, not the guest ones)
pop/push(Use the host segment descriptors for the size of the operands and the stack pointer)
call FAR,jump FAR,int,retf(Transfer control according to the host descriptors tables)
str(Set the task register of the host)
mov <segment registers>(use the host descriptors tables, not the guests ones)
The rationales behind the sensitivity is a work of mine
None of this instruction trap always.
Some, not counting memory access exceptions, never do: pushf, popf, lar, lsl, verr, verw, push, pop.
Some trap only if the host has configured them to do so: smsw, sgdt, sidt, sldt, str.
This is likely to not be what Tanenbaum intended to say though.
Some trap almost certainly but some values can make them work: call FAR, jmp FAR, retf, int.
This is probably what Tanenbaum meant.
Put in simple words, an instruction like jmp FAR 08h:00h is trying to access the "code labelled by the number 08h".
This may or may not succeed depending on what restriction the host put on the "label" 08h.
Most of the label are not accessible and they will trap, but some can work.
The same is true for call and retf.
int usually doesn't trap but that again depends on the OS configuration.
In general instructions that depends on the host's system structures can trap for certain values but not for other.
No harm can be done, even if the instruction turn out to execute successfully, but it cannot be executed directly in a virtualization context.