I'd like to monitor the usage of our Lambdas' ephemeral storage, but I don't want to use the UI tools like Lambda Insights. We currently have a log-scanning python script set up that reads the logs to attain runtime, memory use/limit, etc., but also to find specific things in the logs like code warnings, errors, and other easter eggs we drop in our own logs.
Is there any way we could monitor how much of the allocated storage is used by each Lambda from the logs? I know there's nothing in the logs currently, wondering if there's some code that will dump the ephemeral usage into the log so we can pick it up from there.
Found out that the psutil library can be leveraged in a Lambda to attain this value. Tested and seems to work perfectly. I set the ephemeral memory limit for my Lambda, generated fake files, loaded them into the /tmp/ folder until the Lambda threw an error, and found that this reports the usage correctly:
import psutil
def get_memory_usage():
"""Returns the current memory usage of the Lambda function in MB"""
return round(psutil.Process().memory_info().rss/(1024*1024),1)
def lambda_handler(event,context):
print(f'starting eph memory use: {get_memory_usage()}MB')
#run code, etc
print(f'ending ephemeral memory use: {get_memory_usage()}MB')
return 'complete'
NOTE: if you run this script on your local machine, it will report other memory usage. On Lambda, it reports ephemeral usage.
Also note that the import statements you run will use the ephemeral so that's why I show a starting memory usage and a final, so that I can see what my imports are using and what my code is using. Obviously the highest number is the one that actually counts towards the set limit.
Also, for the sake of log parsing, I only use the full word ephemeral in the final value so my log parsing script can easily find it.