Why this won't cause page fault?

I use BOCHS to emulate Intel-80386 and try to write an operating system, but when I use an element of PCB after destroying PCB, it won't cause a page fault. It confused me for a few days. Maybe because of cache or something? Can anyone help me please?

Here is my project github link:https://github.com/Ares-bit/xos

This is my problem:

//release a process
void thread_exit(struct task_struct* thread_over, bool need_schedule)
{
    intr_disable();
    thread_over->status = TASK_DIED;

    if (elem_find(&thread_ready_list, &thread_over->general_tag)) {
        list_remove(&thread_over->general_tag);
    }

    if (thread_over->pgdir) {
        mfree_page(PF_KERNEL, thread_over->pgdir, 1);
    }

    list_remove(&thread_over->all_list_tag);

    //I release a PCB physical page here, so the whole PCB can not be accessed anymore
    if (thread_over != main_thread) {
        mfree_page(PF_KERNEL, thread_over, 1);
    }

    //But it still can access pid of PCB without PAGE FAULT
    release_pid(thread_over->pid);

    if (need_schedule) {
        schedule();
        PANIC("thread_exit: should not be here\n");
    }
}

This is mfree_page:

//release a physical page
void mfree_page(enum pool_flags pf, void* _vaddr, uint32_t pg_cnt)
{
    uint32_t pg_phy_addr;
    uint32_t vaddr = (int32_t)_vaddr, page_cnt = 0;
    ASSERT(pg_cnt >=1 && vaddr % PG_SIZE == 0);

    pg_phy_addr = addr_v2p(vaddr);

    ASSERT((pg_phy_addr % PG_SIZE) == 0 && pg_phy_addr >= 0x102000);
    if (pg_phy_addr >= user_pool.phy_addr_start) {
        vaddr -= PG_SIZE;
        while (page_cnt < pg_cnt) {
            vaddr += PG_SIZE;
            pg_phy_addr = addr_v2p(vaddr);
            ASSERT((pg_phy_addr % PG_SIZE) == 0 && pg_phy_addr >= user_pool.phy_addr_start);
            pfree(pg_phy_addr);
            page_table_pte_remove(vaddr);
            page_cnt++;
        }
        vaddr_remove(pf, _vaddr, pg_cnt);
    } else {
        vaddr -= PG_SIZE;
        while (page_cnt < pg_cnt) {
            vaddr += PG_SIZE;
            pg_phy_addr = addr_v2p(vaddr);
            ASSERT((pg_phy_addr % PG_SIZE) == 0 && pg_phy_addr < user_pool.phy_addr_start \
                                                && pg_phy_addr >= kernel_pool.phy_addr_start);
            pfree(pg_phy_addr);
            page_table_pte_remove(vaddr);
            page_cnt++;
        }
        vaddr_remove(pf, _vaddr, pg_cnt);       
    }
}

Here is page_table_pte_remove:

//remove pte from page table
static void page_table_pte_remove(uint32_t vaddr)
{
    uint32_t* pte = pte_ptr(vaddr);
    *pte &= ~PG_P_1;
    //invalidate TLB
    asm volatile("invlpg %0" : : "m"(vaddr) : "memory");
}

I've used invalidate TLB by "invlpg" to clean TLB, but it still can access pid, why it won't cause a PAGE FAULT?

Solution

asm volatile("invlpg %0" : : "m"(vaddr) : "memory");

is incorrect. To understand why, we need to know something about the invlpg instruction as well as gcc inline asm, both of which are somewhat counterintuitive in this particular example.

invlpg is encoded with a normal x86 memory operand, like invlpg (%eax), where the effective address of the memory operand is the address whose page table entry is to be invalidated. (In Intel syntax, this would be invlpg [eax].) So this example uses the contents of %eax as an address and invalidates that page. An instruction like invlpg (%eax, %ebx, 4) (Intel invlpg [eax+ebx*4]) would multiply ebx by 4, add eax, and use the result as the address to be invalidated. Note that despite how the instruction might look, the address is not loaded from memory. invlpg doesn't do any memory access at all; it only affects the TLB.

Now when you use an m operand in gcc inline asm, the compiler ensures that the specified expression (here vaddr) is stored at some address in memory, ensures that this address can be represented via an effective address operand, and substitutes %0 with the string for that operand. Since vaddr is already present on the stack (having been passed to this function as an argument), the compiler simply substitutes %0 with a memory operand specifying this location on the stack, such as 4(%esp), and the resulting asm is something like invlpg 4(%esp) (try on godbolt). (The Intel syntax equivalent would be invlpg [esp+4].) Or, possibly invlpg 8(%ebp) if using a frame pointer.

So in essence, you are not invalidating the page whose address is the value of vaddr; you're invalidating the page whose address is the address of vaddr, i.e. the page on the stack where that argument is stored. This is not what you want.

There are two straightforward ways to fix it. One is simply to have the compiler put the value of vaddr into a register (using an r operand), and then manually specify a memory operand using that register:

asm volatile("invlpg (%0)" : : "r"(vaddr) : "memory");

This emits mov 4(%esp), %eax ; invlpg (%eax). Try on godbolt.

Another is to keep using an m operand, but specify an expression whose address (not value) is the address to be invalidated. This means converting vaddr to a pointer and dereferencing it.

asm volatile("invlpg %0" : : "m"(*(char *)vaddr) : "memory");

This also emits mov 4(%esp), %eax ; invlpg (%eax). Try on godbolt. Despite how it may look, this does not actually access the memory at address vaddr, so it's safe regardless of the current status of that page. This second version gives the compiler a little more flexibility to use other addressing modes in constructing the operand, but it's unlikely to be relevant in this setting.