I have a process that consumes a lot of memory on startup, but frees most of that memory after the process is bootstrapped. I see the following in the TCMalloc stats printed afterwards:
#012MALLOC: 16635888 ( 15.9 MiB) Bytes in use by application
#012MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
#012MALLOC: + 20007352 ( 19.1 MiB) Bytes in central cache freelist
#012MALLOC: + 26367680 ( 25.1 MiB) Bytes in transfer cache freelist
#012MALLOC: + 7030680 ( 6.7 MiB) Bytes in thread cache freelists
#012MALLOC: + 1978560 ( 1.9 MiB) Bytes in malloc metadata
#012MALLOC: = 72020160 ( 68.7 MiB) Actual memory used (physical + swap)
#012MALLOC: + 239607808 ( 228.5 MiB) Bytes released to OS (aka unmapped). ***
#012MALLOC: = 311627968 ( 297.2 MiB) Virtual address space used
Here we see that there's ~228 MB "released to the OS", but then it also shows that this is still part of the process' virtual address space. This is corroborated by the fact that the VSZ stat seen with ps aux remains high after this logging is seen.
When I run the same program with strace, I can see that: all of the calls to brk are increasing, and that the calls to munmap (in terms of sizes) do not add up to the amounts mmap-ed (and showed above). What's worse is that VSZ increases slowly over time as my process does background work, and the logging shows that (***) increases correspondingly when these stats are logged.
So my questions are: What does this value represent? Has my memory been freed or not? What can I do to prevent this consumption from growing?
What does this value represent?
It represents the amount of memory that TCMalloc has told the system that it does not need and that the system may use for other purposes.
Has my memory been freed or not?
No. The OS has decided that making it free just to have to make it used again when the program needs it is a waste of effort and has decided instead to switch it directly from one use to another use in a single step rather than going through twice the effort of making it free just to have to make it un-free to use it.
What can I do to prevent this consumption from growing?
Why would you want to? It just makes it easier if the program needs the memory later, minimizes contention on the system's free list, and has no harmful effects at all. TCMalloc has told the OS (via madvise(DONTNEED)) that the OS may recover the memory and the OS has made the decision that it's not a good idea to make it free just to have to make it used again when it's needed. Do you have some good reason to think the OS is wrong?
It's much easier to just directly transition memory from one us to another in a single step than go through the two steps of making it free just to have to make it used again. The free list can get contended under load and it's much simpler not to use it.
You can force the OS to free it by running some program that consumes a lot of memory and then terminates. That will force the OS to transition the memory to that process and then free it when that process terminates. But this would provide no benefit at all and would be a lot of effort just to eventually increase contention in the memory manager. There is no issue here.