I'm trying to set up some code to monitor how much RAM my jupyter notebook is using. This is in a virtual machine environment (an AWS cloud server). I have code that can easily use up over 50 GB of RAM. I need to monitor how much RAM I'm using so the code will automatically shut itself off if it starts to get close to the max RAM; eventually, this will get written so the code parses the job out in little chunks, but that's for future me to deal with. Right now, I just want to know how to monitor the RAM usage of the jupyter notebook. The methods I have tried so far give outputs that don't make sense.
Here's the code for the two different methods I have tired so far to check the RAM.
Version 1
import psutil
# This is not correct!
print('RAM Used [GB]:', psutil.virtual_memory()[3]/1000000000)
print('RAM Total [GB]:', psutil.virtual_memory()[0]/1000000000)
Puts out:
RAM Used [GB]: 8.34738176
Should be ~2.82 GB
RAM Total [GB]: 32.99915776
Should be ~16.00 GB
Version 2
import os
# Getting all memory using os.popen()
total_memory, used_memory, free_memory = map(
int, os.popen('free -t -m').readlines()[-1].split()[1:])
# Memory usage
print("RAM memory % used:", round((used_memory/total_memory) * 100, 2))
print(f"total_memory = {total_memory}, used_memory = {used_memory}, free_memory = {free_memory}")
Puts out:
RAM memory % used: 25.29
Should be ~2.8/16 or 17.5%
total_memory = 31470, used_memory = 7960, free_memory = 6575
should be ~16GB, ~2.8 GB, ~13.8 GB
Screencap of the code and actual memory usage (according to jupyter notebook)
My first thought is that this was somehow reading the RAM usage for my laptop, but even then, it's not right (the usage was too low). What am I doing wrong here? Is there another way to measure the RAM that a jupyter notebook is using?
UPDATE: I've tried getting the RAM using a different sized Jupyter notebook, and the results were pretty odd.
I resized the notebook to 8 GB of RAM, and the used and total RAM values were close. Still not correct, but good enough for what I would need to do.
Redid it with a 16 GB RAM run, and I got the crap RAM numbers again.
screenshot of 16 GB RAM re-run
Sooo, it's starting to seem like determining how much RAM a jupyter notebook is using via programmatic methods is impossible.
On AWS, Jupyter Notebook servers are sometimes initialized as shared machines. In this specific case, it was a single server for two users, splitting 32 GB of RAM into two 16 GB sets, one for each user. To my knowledge, there is not way to programmatically get that info from within the Jupyter Notebook.
So, the solution is to have the user specify their RAM allocation. Annoying, but it works.
