I know about the system calls that OS provides to protect programs from accessing other programs memory. But that can only help if I have used the system call library provided by OS. What if I write a assembly code myself that sets CPU bit for kernel mode and executes a privileged instruction ( let's say modify OS' program segment in memory ). Can OS protect against that ? P.S. Out of curiosity question. If any good blog or book reference can be provided, that would be helpful as I want to study OS in as much detail as possible.
This blog clearly explains what my confusion was. http://minnie.tuhs.org/CompArch/Lectures/week05.html Even though user programs can switch to kernel mode, but they have to do it through a interrupt instruction ( int in case x86) and for this interrupt, the interrupt handler is written by the OS. ( probably when it was in kernel mode at bootup time). So this way all priviliged instructions can only be executed by the OS code only.