Background: I have a python script which sends & recieves audio using the aiortc library, and it generally works well (which is impressive considering I'm by no means a software developer). However, I've noticed that over multiple days the memory usage of this script slowly balloons out to fill all available memory on the host server. This only happens when the audio RTC streams are running, which rules out other parts of my code and has led me to focus exclusively on the aiortc library itself.
Interestingly, the usual recommended python memory tools, tracemalloc & muppy, do not show this large memory usage anywhere - the largest items in memory they can find is typically a couple MB, when the script as viewed via htop is using upwards of 7 GB. Example output from muppy:
types | # objects | total size
============================ | =========== | ============
str | 40749 | 6.00 MB
dict | 13615 | 4.83 MB
code | 12379 | 2.15 MB
tuple | 30681 | 1.91 MB
type | 2014 | 1.77 MB
set | 895 | 570.29 KB
list | 2457 | 526.35 KB
abc.ABCMeta | 363 | 392.88 KB
weakref | 4387 | 308.46 KB
builtin_function_or_method | 4252 | 298.97 KB
int | 6281 | 174.71 KB
wrapper_descriptor | 2461 | 173.04 KB
getset_descriptor | 2432 | 152.00 KB
tracemalloc.Statistic | 2337 | 127.80 KB
frozenset | 419 | 124.13 KB
This leads me to believe that the source of the leak is something lower level in the non-python libraries the aiortc library is calling. I'm not the only one who's reported memory leak issues with the library, but so far nobody has managed to come up with any smoking guns.
I'm curious what the recommended course of action would be to further dig into the memory usage issues I'm seeing, when the normal python memory profiling tools aren't working. Are there lower-level python memory profiling tricks, or do I need to go deeper into external memory profiling tools?
Perhaps jemalloc is worth a try?