I am working on a project where a dynamic library (.so) is injected in some target program at runtime (dynamic instrumentation). The library handles its own memory using mmap/munmap. For security reasons, it is required that some mapped region in the library only be writable through the exposed APIs from the library.
What we've done is toggle the write flag of the memory region using mprotect and PROT_WRITE at the prologue/epilogue of the library functions, e.g.:
void foo(void) {
mprotect(addr, PAGE_SIZE, PROT_READ | PROT_WRITE);
...
...
mprotect(addr, PAGE_SIZE, PROT_READ);
}
This works fine for single threaded applications. For multi-threaded applications, other tasks in the same process might be able to write to the mapped library region if a context switch (to a different task in the same process) takes place after the PROT_WRITE was set (so the memory is writable) and before it was cleared.
So, the question is: Is it possible to disable other tasks in the process until foo returns? If not, how do you suggest working around this?
Your security model is not valid. If your library code can mprotect these regions writable, so can any other part of the program. You cannot prevent this introspectively. Such security properties can only be achieved with some sort of outside supervision, either directly by the kernel or by a tracing process (e.g. using traditional ptrace or seccomp tracing).
If you don't actually need security properties, but just want reliable trapping of writes by already-trusted code, you could achieve something like this by keeping track of all your threads, signaling them all with pthread_kill, and having the signal handler block until the signaling thread allows them to continue.