pythonlinuxperformancesignalspopen

How to measure a server process maximum memory usage from launch to SIGINT using /usr/bin/time in Python


I have a server program (let's call it my-server) I cannot alter and I am interested in measuring its maximum memory usage from when it starts to when a certain external event occurs.

I am able to do it manually on my shell by:

I'd like to do this programmatically using Python. If not possible I'd like to find an alternative simple way to achieve the same result.

I tried launching the same command as a subprocess and sending an INT signal to it:

import signal
import subprocess

proc = subprocess.Popen(
    ["/usr/bin/time", "-f", "%M", "my-server"],
    stderr=subprocess.PIPE,
    stdout=subprocess.PIPE,
    text=True
)
# ...wait for external event
proc.send_signal(signal.SIGINT)
out, err = proc.communicate()  # both out and err are ""

however both out and err are empty.

The same script works fine for processes that terminate:

proc = subprocess.Popen(
    ["/usr/bin/time", "-f", "%M", "echo", "hello"],
    stderr=subprocess.PIPE,
    stdout=subprocess.PIPE,
    text=True
)
out, err = proc.communicate()  # out is "hello", err is "1920"

but for processes that need to be terminated via signal I am not sure how to retrieve stderr after termination was issued. (or even before it, despite not useful)

The following somewhat equivalent example may be useful for testing:

terminate-me.py

import signal
import sys

def signal_handler(sig, frame):
    print('Terminated', file=sys.stderr) # Need to read this 
    sys.exit(0)

signal.signal(signal.SIGINT, signal_handler)
print('Send SIGINT to terminate')
signal.pause()

main.py

import signal
import subprocess

proc = subprocess.Popen(
    ["python", "terminate-me.py"],
    stderr=subprocess.PIPE,
    stdout=subprocess.PIPE,
    text=True
)
# No need to wait for any external event
proc.send_signal(signal.SIGINT)
out, err = proc.communicate()
print(out)
print(err)

Solution

  • It turns out I misunderstood /usr/bin/time's behavior.

    I thought that when receiving an INT signal time would relay it to its sub-process being measured, waited for it to terminate, and then terminated normally outputting its results.

    What happens instead is that time simply terminates before outputting any result.

    When executing within a shell, pressing CTRL + C actually sends an INT signal to both, but the sub-process terminates before time does, so time is able to complete its execution and it appears as if time awaited the sub-process.

    Therefore in my script I was able to make it work by only sending the INT signal to the sub-process: (actually, to all of its sub-processes for simplicity)

    import signal
    import subprocess
    import psutil
    
    proc = subprocess.Popen(
        ["/usr/bin/time", "-f", "%M", "my-server"],
        stderr=subprocess.PIPE,
        stdout=subprocess.PIPE,
        text=True
    )
    
    # Wait for external event...
    
    children = psutil.Process(proc.pid).children()
    for child in children:
        child.send_signal(signal.SIGINT)
    out, err = proc.communicate()