In my last job interview I was asked what seems to be a very straight forward simple question:
Q: In which library syscall (The one is kernel space not the wrapper in libc) is implemented?
A: I answered <unistd.h>
The interviewer told me that it's wrong and he is asking in which library it's implemented not in which header file it's declared.
Why is my answer false, what's the correct answer?
I searched the web for hours and nothing found at all, even writing man 2 syscall in shell gives:
#include <unistd.h>
#include <sys/syscall.h> /* For SYS_xxx definitions */
long syscall(long number, ...);
syscall is a wrapper that actually loads the register and executes the instruction syscall on 64 bit x86 or int 80h or sysenter on 32 bit x86 and it is part of the standard library.
example:
syscall:
endbr64
mov rax,rdi
mov rdi,rsi
mov rsi,rdx
mov rdx,rcx
mov r10,r8
mov r8,r9
mov r9,QWORD PTR [rsp+0x8]
syscall
So the answer is that that syscall function is in the glibc.
In the kernel in the assembly file the syscall,sysentry instruction entry or int 80h interrupt handler (depending on the system implementation) does some stack magic, performs some checks and then calls the function which will handle the particular system call. Addresses of those functions are placed in the special table containing function pointers. But this part is very hard to be called the "library".