linuxx86virtual-machinevirtualizationapic

Understanding the virtual APIC page for x2APIC


I am writing a VMM, and I'm trying to support virtual accesses to the x2APIC's registers by a guest OS running in VMX non-root mode.

I want to start off by doing something simple, such as reading the local APIC ID from within the guest OS. I've tried adding support for this in my VMM, but the value I read seems to be incorrect.

Unfortunately, I can't seem to find a lot of information online about the virtual APIC page. I've read through chapter 29 of the Intel manual (APIC Virtualization and Virtual Interrupts), and here's what I'm doing:

  1. In the secondary processor-based VM-execution controls, I set the following bits to 1: (I'm setting bit 9 below since I ultimately want to support posted IPIs)

    1. SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE (bit 4)
    2. SECONDARY_EXEC_APIC_REGISTER_VIRT (bit 8)
    3. SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY (bit 9)
  2. In the MSR bitmap, I disable intercepts for 0x802, which is the local APIC ID register.

  3. In my guest OS, I use rdmsr to read 0x802.

I perform step 3 on two threads that are pinned to different cores. They both read the value 2621447225 from the register. This seems incorrect since the threads are pinned to different cores and should therefore read different local APIC IDs (and the number 2621447225 is really big). What am I doing incorrectly?

Here's some additional information for your reference:

In section 29.5 (Virtualizing MSR-based APIC Accesses) of the Intel manual, it says:

If “APIC-register virtualization” is 1 and ECX contains a value in the range 800H–8FFH, the instruction reads the 8 bytes from offset X on the virtual-APIC page into EDX:EAX, where X = (ECX & FFH) « 4. This occurs even if the local APIC is not in x2APIC mode (no general-protection fault occurs because the local APIC is not in x2APIC mode).

The X offset makes sense to me: the MSR address 0x802 will be 0x2 when AND'd with 0xFF, and 0x2 will become 0x20 when left-shifted 4 bits. 0x20 is the offset within the physical APIC's page if you were accessing an xAPIC through its memory-mapped registers. 8 bytes (i.e. 64 bits) are then read, so the lower 32 bits are the local APIC ID for x2APIC.


Solution

  • I was able to figure this out with the help of @prl. I had to allocate a virtual-APIC page myself for each core and then initialize each page individually with its associated core's local APIC ID.

    I then added the physical address of the page to the VMCS (there's a constant defined in the Linux kernel called VIRTUAL_APIC_PAGE_ADDR that contains the offset within the VMCS). I didn't realize that I had to initialize the page since it wasn't done automatically.

    Edit: I implemented a working virtualization system on Linux with support for x2APIC virtualization and posted interrupt processing, and wrote about those two topics in this document. The document should be fairly straightforward, and it contains a link to my implementation.