I know that operating system restricts the access to kernel code & data by using segmentation and privilege level. However, users can change the segment register value and seems that we can access the kernel data if the following code executes successfully:
mov eax, 0x10
mov es, ax #point selector to the item 2 in GDT with RPL 0, which is the data segment
les bx, [0]
So I'm wondering what's the mechanism that prevents this code from executing successfully?
The mov es, ax instruction will either cause a general protection (#GP) fault because current privilege level (CPL) is greater than the descriptor's privilege level (DPL), or the requested privilege level (RPL) will be ignored because its not numerically higher than the DPL. In your example, since it's running in a user mode, the CPL is 3. This means that he DPL of the descriptor also has to be 3 or the the instruction will fault. If the DPL is 3 then then there won't be a fault, but the RPL is effectively ignored because it can't be higher than the DPL.
(Note that segment privilege level checks are only performed when a segment register is loaded, so it's only the mov es, ax instruction that can crash because of them.)
The documentation for the MOV instruction in the Intel Software Developer's Manual explains when it will cause #GP fault when loading a segment register:
IF DS, ES, FS, or GS is loaded with non-NULL selector THEN IF segment selector index is outside descriptor table limits OR segment is not a data or readable code segment OR ((segment is a data or nonconforming code segment) AND ((RPL > DPL) or (CPL > DPL))) THEN #GP(selector); FI; IF segment not marked present THEN #NP(selector); ELSE SegmentRegister := segment selector; SegmentRegister := segment descriptor; FI;
The behaviour of the highest of the DPL and RPL being used is documented in the Intel SDM Volume 3, "5.5 PRIVILEGE LEVELS":
- Requested privilege level (RPL) — [...] Even if the program or task requesting access to a segment has sufficient privilege to access the segment, access is denied if the RPL is not of sufficient privilege level. That is, if the RPL of a segment selector is numerically greater than the CPL, the RPL overrides the CPL, and vice versa. [...]
The RPL field of a selector only allows the effective privilege level to be increased to a numerically higher, or less privileged level than the DPL. It has no effect if you set it to a numerically lower level.
In other words, if selector 0x10 refers to a kernel mode data segment (DPL = 0) then your code will crash. If selector 0x10 is a user mode data segment (DPL = 3), it's treated the same as if you used 0x13 (RPL = 3) instead.
Note that in practice this doesn't really matter much, as all modern operating systems use a flat segment model were every segment has a base of 0 and can access the entire linear address space. User mode code isn't actually restricted from accessing kernel code and data through segment checks, but through page protections. These use only the CPL to determine whether access should be granted to supervisor mode (kernel) pages.