Suppose I've disabled all the oom related features(no OOM killer). A process has occupied all the available memory and it's still trying to read some bytes from a mmapped disk file, which is not in any page caches. Will this process get a OOM signal so that it can reactively release some memory and retry later?
It depends on many conditions.
1)how you disable oom-killer?
Let's suppose you write 2 to /proc/sys/vm/overcommit_memory, which mean:
2: always check, never overcommit (see man 5 proc)
After that you called mmap.
2)What flags do you use to in "mmap"?
Let's suppose you use MAP_NORESERVE, in another case (without MAP_NORESERVE) mmap just return error, if not enough physical memory.
3)What is your overcommit_ratio value? Let's suppose it is not zero, if it is zero then mmap return error, and we cann't be in situation where "no pages for file".
If all these suppositions are true, then you come to: mm/oom_kill.c::pagefault_out_of_memory,
and again new condition:
4)May be we in cgroup with oom disable?
If yes, then we just go to sleep.
And at last oom-killer is called
About disable oom
And by "we just go to sleep", do you mean a system pause?
See linux-source/Documentation/cgroups/memory.txt:
If OOM-killer is disabled, tasks under cgroup will hang/sleep in memory cgroup's OOM-waitqueue when they request accountable memory.
So process that part cgroup with disabled oom-killer go to sleep, not all system.
Is it possible that we, the userland process handle this pagefault?
1)It is possible to do this: https://lwn.net/Articles/550555/
2)Or you can just watch oom-killer events.
See again linux-source/Documentation/cgroups/memory.txt:
memory.oom_control file is for OOM notification and other controls.
Memory cgroup implements OOM notifier using the cgroup notification API (See cgroups.txt). It allows to register multiple OOM notification delivery and gets notification when OOM happens.
To register a notifier, an application must:...