A friend of mine is working on a library for memory-constrained Linux systems. He’s proposing using shm_open to allocate multiple reasonably-sized (16MB) amount of memory for interprocess communication across a number of different programs on the user’s machine.
The concern that came up is that if there are lots of buffers allocated (say, 128), then 128 × 16MB may be a nontrivial fraction of the available system memory, and there’s a good chance that not much of the reserved memory would actually get used. For example, if only, say, 128K of that memory in each buffer was actually used for anything, this approach would use around 1/128 of the memory reserved. Due to the anticipated access patterns, it’s likely that only small regions of each buffer would be “hot” at any given time.
I consulted the man pages for shm_open, which mentioned that on Linux specifically, the implementation uses tmpfs for the allocated memory. The man page for tmpfs in turn says that the memory allocated can be paged out if there’s physical memory pressure on the machine. It also says that only space needed to store the used contents of the file system will be allocated.
By my reading of this, I’m assuming that the following are true:
Using shm_open to allocate 16MB of space won’t necessarily immediately consume 16MB of physical memory on the machine, since most of the file system will be zero pages, which the OS will lazily allocate. The space used will be proportional to the amount of data written.
If there is little physical RAM left on the machine, the OS is permitted to page out sections from the shared memory buffer. Moreover, if only certain sections of the buffers are going to be accessed at any one time, it would not be unreasonable to assume that those regions - and probably not other regions - would be paged in at a given time.
Are these assumptions reasonable? This is something that in principle could be tested empirically, but the concern is that we’d get into issues of the form “yes, that works on your system, but that’s not generally true of other Linux installs.”
The only way that user-mode application programs can generally reserve physical memory is by using the mlock family of system calls; the process has to either have the CAP_IPC_LOCK privilege or they're limited to RLIMIT_MEMLOCK bytes.
Shared memory can be locked into RAM with mlock(), but it's not done automatically, and there's no reason why it would need to be. It's just shared virtual memory.