x86kernelosdevmemory-segmentationprotected-mode

How did executable stacks work in x86 protected mode?


Section 4.7.2 of the the AMD64 Developer Manual Volume 2 (System Programming) which describes Code-Segment Descriptors in legacy modes states:

Code segments establish the processor operating mode and execution privilege- level. The segments generally contain only instructions and are execute-only, or execute and read- only. Software cannot write into a segment whose selector references a code-segment descriptor.

If code segments are the only executable segments, and they cannot be written to, how did executable stacks on 32 bit systems work?


Solution

  • Code segments are executable (but not writable). Stack segments must be read/write (and aren't executable).

    Each segment has a base address and a limit. Software provides offset/s into segment/s (e.g. the instruction pointer is an offset into the area described by the cs segment, the stack pointer is an offset into the area described by the ss segment); where the CPU checks that the offset is within the segment's limit, then adds the base address of the segment to the offset to determine the (linear) address.

    If the code segment and stack segment refer to the same memory (e.g. have the same base address and limit); then that same memory can be executed (using the code segment) and read/write (using the stack segment).

    Note that most operating systems don't/didn't use segmentation (they set all segment base addresses to zero and all segment limits to "max" so that it behaves like there's no segments); and "all segment refer to the same memory" is very common. For protection, most operating systems use paging only.