coperating-system

System call numbers, trap handling, and the C library’s role in system calls

While reading the OSTEP book, I had a question.

From the book:

To specify the exact system call, a system-call number is usually assigned to each system call. The user code is thus responsible for placing the desired system-call number in a register or at a specified location on the stack; the OS, when handling the system call inside the trap handler, examines this number, ensures it is valid, and, if it is, executes the corresponding code. This level of indirection serves as a form of protection; user code cannot specify an exact address to jump to, but rather must request a particular service via number.

I’m trying to better understand how system call numbers work within the context of system calls and trap handling. Here’s what I understand so far:

  1. The system call number is used like a key or index in a trap table, allowing the CPU to jump to the correct trap handler.
  2. Inside the trap handler, the operating system validates the system call number.
  3. The C library’s system call API stores the system call number in a register or on the stack before invoking the trap.

Is my understanding accurate? Could someone clarify or provide additional details, especially regarding the role of the C library and the validation step?

Solution
    1. The system call number is used like a key or index in a trap table, allowing the CPU to jump to the correct trap handler.

    No.

    A "trap table" can be found in CPU architectures that support a TRAP N instruction where N is the trap number. In such architectures, the trap table maps trap numbers to addresses of trap handlers. An operating system typically defines one trap number for offering system calls.

    Suppose we have a hypothetical CPU which supports multiple trap handlers, and an operating system which uses TRAP 42 for system calls.

    The system call number is something you have to set up before invoking the operating system trap. For example:

    LOAD R1, system_call_number
TRAP 42

    So, the CPU uses that 42 to look up the trap handler in the trap table, and jumps to it. This will be the operating system trap handler. Within that handler, the operating system uses the value of R1 to look up, in another table, (call it the "system call" table,) the address of the system function, and jump to it.

    Note that the process is very similar if the CPU supports a parameterless instruction for invoking the operating system:

    LOAD R1, system_call_number
SYSCALL

    In this case, there is only one handler, and no trap table. However, it is still possible to make many different system calls, so the mechanism with system call numbers, resolvable via a system call table, is still there.

    1. Inside the trap handler, the operating system validates the system call number.

    Uh, sure. If the system call number does not correspond to an actual valid system function, the operating system will probably do bad things to your process.

    1. The C library’s system call API stores the system call number in a register or on the stack before invoking the trap.

    I do not know what you mean by "The C library’s system call API". In any case, anyone interested in invoking system calls must store the system call number somewhere before invoking the operating system. The operating system defines where the number should be stored. It is usually in a register.