I'm running attached code in 64-bit system, so I expect for this buffer bytes (21B), I'd be able to allocate memory, but calloc fails to do that and so it returns a NULL pointer. For smaller buffer bytes like 20 B, it will be fine but breaks at 21 B and as I checked for different machines, this breaking point can vary. I just need to know what should I check in that system that it's inhibiting me from this memory allocation?
#include <unistd.h>
#include <cstdio>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <iostream>
int main(int argc, char* const argv[])
{
size_t bufferBytes = 21000000000;
// size_t bufferBytes = 20000000000;
// Use words instead of bytes to ensure 8-byte alignment.
size_t bufferWords = bufferBytes / 8;
auto m_buffer = reinterpret_cast<char *>(calloc(bufferWords, 8));
if(m_buffer==NULL)
{
std::cout<<"Not a valid pointer"<<std::endl;
}
}
The classic problem of "how much memory can I allocate with malloc" depends on multiple factors such as your OS, your library implementation of malloc/calloc and how much RAM you have.
Generally, however, for such huge allocations, library implementations bypass the heap entirely and go straight for the OS services (mmap on Linux, VirtualAlloc on Windows). It is up to the OS to decide what limit they impose on these allocations. For example, my Windows machine tops out at ~52GB for a single calloc (with 16GB of physical RAM), but this limit is dynamic (if RAM is heavily used by various processes, this limit will be lower).
The virtual address space of a 64-bit systems is (for now) really big. However, this is not true for the physical address space, where most consumer computers have 8-32GB of RAM.
When you demand such a large allocation, your OS doesn't actually reserve any physical memory for you, but instead only marks a certain area in your virtual address space as usable. When you try to access that memory, that's when the OS will actually allocate it for you. You can see this by checking your process in the Task Manager (or htop on Linux). Add a while(1); at the very end of the program and you will see that your program barely uses any memory.
Once you start using more memory than you actually have RAM, the OS will start swapping it to disk (making your computer run very slowly), so that it can make space in RAM. However, it can only do this up to a certain point. It can't use hundreds of gigabytes on your hard drive for swap. The OS will therefore top out at a certain size (see setrlimit on Linux), so that it doesn't promise you memory that it might not be able to give you.