cmemory-managementlinux-kernelkmallocvmalloc

Memory Mapping in Linux Kernel - use of vamlloc() and kmalloc()

Considering a 32bit x86 Linux system with 4 GB of RAM memory, So as described in books as well as on many forums that the Memory mapping would be as follows:

  1. Kernel logical address - upto 896 MB - Which is one to one mapped and can be allocated using kmalloc().
  2. Kernel virtual address - 128MB (above 896MB - kernel logical address) - allocated using vmalloc() and allocates Virtually contiguous but physically(scattered within RAM) non-contiguous memory pages.

Few points that I am not able to fully understand and need clarity on.

  1. My understanding is that When kmalloc() is used to allocate memory, It always comes from the 0 to 896MB within the RAM and not beyond that.

  2. When we use vmalloc() to allocate memory, Does that memory allocated anywhere from 896MB to 4GB range within the RAM ? or it is allocated only from 896MB upto 1GB range within RAM?

  3. When we say that kernel has only 1GB of virtual address space, Does that meant that the kernel can not access the RAM beyond 1GB ? If it can then how it is done ? Does the 128MB of kernel virtual address space are used for this purpose ?

Please help.

Solution

  • In theory there are 3 different "memory managers". One manages physical RAM (mostly keeping track of pages of free physical RAM), one manages virtual space (what is mapped into each virtual address space where, working with fixed size pieces - the page size), and the third manages "heap" (allowing a larger area of the virtual address space to be split up into arbitrary sized pieces).

    Originally; the Linux kernel tried to use its kernel "heap" to manage all 3 of these very different things. By linearly mapping "all RAM" into kernel space they bypass the need for managing the kernel's virtual memory and end up with a simple relationship between virtual addresses in kernel space and physical addresses (e.g. "physical = virtual - base"), and by allocating "heap" you also allocate physical memory.

    This was fine originally because at the time computers rarely had more than 128 MiB of RAM (and Linus didn't expect the kernel to exist for very long, as GNU were planning to switch to Hurd "soon"), and kernel space was significantly larger than "all RAM". As the amount of RAM increased it became a problem -"all RAM" became larger than kernel space, so "use heap to manage 3 very different things" couldn't work.

    Of course once it became a problem a lot of the kernel's code dependent on "kmalloc to allocate physical memory", making it too hard to fix the problem. Instead, they split physical memory into 2 zones - one zone that would be managed by "kmalloc" and another zone that is managed by "vmalloc"; then changed pieces of the kernel to use "vmalloc" instead of "kmalloc" where it's easy make those changes.

    1. My understanding is that When kmalloc() is used to allocate memory, It always comes from the 0 to 896MB within the RAM and not beyond that.

    Yes; this is the first zone of physical memory, which fits into the kernel space mapping that "kmalloc" uses.

    1. When we use vmalloc() to allocate memory, Does that memory allocated anywhere from 896MB to 4GB range within the RAM ? or it is allocated only from 896MB upto 1GB range within RAM?

    It would be allocated from any RAM that is not in the first zone (anywhere in "896MB or higher" range).

    1. When we say that kernel has only 1GB of virtual address space, Does that meant that the kernel can not access the RAM beyond 1GB ? If it can then how it is done ? Does the 128MB of kernel virtual address space are used for this purpose ?

    Of the kernel's 1 GiB of virtual space; some (896MB) will be the linear mapping of the physical address space, some will be memory mapped (PCI) devices, and some will be set aside as an area where dynamic mappings can be made. For "vmalloc" the kernel would allocate physical pages of RAM and then map them into the "dynamic mapping area" (and return a pointer to where it was mapped that has nothing to do with its physical address and breaks the "physical = virtual - base" relationship).

    Note 1: Exact sizes/limits are variable - e.g. kernel can be compiled for "2 GiB / 2 GiB split" where kernel space is 2 GiB (instead of "3 GiB / 1 GiB split"); and the size of the "kmalloc zone" probably depends on various factors (how much space PCI devices need, how much RAM there is, etc) and may be something other than 896MB.

    Note 2: Since introducing "vmalloc" to work around the original problem; computers switched to 64 bit (where "all memory" can/does fit in kernel space again), and "vmalloc" became unnecessary (and probably just falls through to "kmalloc"). However a lot of other changes have occurred (introduction of NUMA, encrypted RAM, non-volatile RAM, ..; plus more security vulnerabilities than any single person can keep track of) so the original design flaw has reached a temporary "bad idea, but still technically not broken if we keep adding work-arounds for security vulnerabilities" stage (until RAM and non-volatile RAM sizes inevitably increase and "vmalloc" becomes needed again at some point in the future - probably in about 30 years).