We are using a python process to manage long running python subprocesses. Subprocesses occasionally need to be killed. The kill command does not completely kill the process, only makes it defunct.
Running the following script demonstrates this behaviour.
import subprocess
p = subprocess.Popen(['sleep', '400'], stdout=subprocess.PIPE, shell=False)
or
p = subprocess.Popen('sleep 400', stdout=subprocess.PIPE, shell=True)
Will create a subprocess.
p.terminate()
p.kill()
does nothing to the process. Demonstrated by ps aux | grep sleep
$ ps aux| grep 'sleep'
User 8062 0.0 0.0 7292 764 pts/7 S 14:53 0:00 sleep 400
The process has not been killed/made defunct. Using the subprocess.call() function with 'kill' and pid as arguments will issue the kill command.
subprocess.call(['kill', str(p.pid)])
This will kill the process but it is now defunct.
$ ps aux | grep 'sleep'
User 8062 0.0 0.0 0 0 pts/7 Z+ 14:51 0:00 [sleep] <defunct>
If the queue is running long enough will it eventually reach its maximum number of processes, or will it eventually reap the defunct processes and be fine?
If the answer is the former, how can I handle defunct processes in python without killing the parent process?
Is there a better way of killing processes?
There are 2 main issues here:
First issue: If you're using shell=True, so you're killing the shell running the process, not the process itself. With its parent killed, the child process goes defunct / isn't killed immediately.
In your case, you're using sleep which is not built-in, so you could drop shell=True, and Popen would yield the actual process id: p.terminate() would work.
You can (and you should) avoid shell=True most of the time, even if it requires extra python coding effort (piping 2 commands together, redirecting input/output, all those cases can be nicely handled by one or several Popen without shell=True.
And (second issue) if the process is still defunct when terminating after that fix, you could call p.wait() (from this question). Seems that calling terminate isn't enough. The Popen object needs to be garbage collected.