I'm checking Wow64 subsystem on Windows 64 bit. As far I can tell on x86_64 processors, in order to run 32-bit applications, it leverages on Long mode compatibility sub-mode. Any system call from 32-bit compatibility mode is handled via a switch mode from current mode to 64 bit mode (switching the CS register from 0x23 selector to 0x33 selector).
I have a pair of questions.
The first is that under x32dbg I cannot see the ntdll 64 bit version loaded in the process's address space.
The second is more general regarding the way the CPU (or actually the CPU cores) performs privilege checks on segment selectors being loaded into segment registers. For instance on far jumps (as in the case of Wow64 transition) the CPU implicitly need to load the new segment code selector 0x33 into the CS register. From Intel SDM it performs a check based on the CPL of the code selector currently loaded in CS and the RPL of the new code segment selector being loaded in CS (0x33) against the DPL of the associated code segment descriptor from in-memory GDT table.
My question is: which other type of checks are performed by CPU/cores during operations that (implicitly or explicitly) access memory in those segments ? Thank you.
On modern x86(-64) OSes, segments are only used as much as required by the ISA. And for CS, for setting the current mode and privilege level. Not for memory protection.
With a flat memory model, CS and DS base = 0 and limit = unlimited, so each "segment" covers all of virtual address space, i.e. segmentation isn't used for memory protection, only for user vs. kernel mode.
The only checking that's relevant is when reloading a segment register (e.g. during a far jump, or on mov ds, eax). Access checking during each load or store in 64-bit mode is just paging.
In 64-bit mode, hardware requires a flat memory model; the base and limit are ignored for CS/DS/ES/SS. In 32-bit legacy mode, mainstream OSes were doing that anyway for years before AMD64. In compat mode, I think non-zero DS / CS / ES / SS bases are possible, but again mainstream OSes don't use them. (CPU hardware special-cases base=0 to simplify address calculations in the AGUs in load/store units. They might still do limit checking, but it always passes when limit=-1 with units of 4K.)
FS and GS segments can still have a non-zero segment base in long mode, used for thread-local storage (TLS). I forget if they can have a limit. (https://wiki.osdev.org/Segmentation). You set their base to a value wider than 32 bits via wrmsr, and newer CPUs have wrfsbase / wrgsbase which the OS can let user-space use to change the bases without a system call. (Unfortunately these instructions only work in 64-bit mode, not in compat sub-mode of long mode.)