I want to register a new software interrupt with a corresponding service routine on Windows 10 x64, is this possible - is there API provided to do this? Do I have to accomplish this manually using undocumented features and would doing so trigger Kernel Patch Protection?
Registering a software interrupt results in the interrupt descriptor table (IDT) being modified for the running operating system. Windows has Kernel Patch Protection (KPP) / PatchGuard which protects kernel code and data structures such as the IDT from being tampered with. This includes preventing the addition of new interrupt descriptor entries, the deletion of them, and modifying the descriptor table in general.
This also means that interrupt descriptor entries cannot be modified. The reason developers would do this is to redirect interrupts to new interrupt service routines, also known as interrupt hooking. This is also prevented because of KPP/PatchGuard.
Generally speaking, for the 64-bit versions of Windows starting from XP, PatchGuard/KPP prevents the IDT from being modified in any way. However, Microsoft has explicitly stated that KPP/PatchGuard is disabled when a kernel debugger is activated. Although, critical kernel code and data structures such as the IDT can still be modified on 32-bit versions of Windows regardless of whether a kernel debugger is present or not.