If starting a subprocess from Python:
from subprocess import Popen
with Popen(['cat']) as p:
pass
is it possible that the process starts, but because of a KeyboardInterrupt caused by CTRL+C from the user, the as p bit never runs, but the process has indeed started? So there will be no variable in Python to do anything with the process, which means terminating it with Python code is impossible, so it will run until the end of the program?
Looking around the Python source, I see what I think is where the process starts in the __init__ call at https://github.com/python/cpython/blob/3.11/Lib/subprocess.py#L807 then on POSIX systems ends up at https://github.com/python/cpython/blob/3.11/Lib/subprocess.py#L1782C1-L1783C39 calling os.posix_spawn. What happens if there is a KeyboardInterrupt just after os.posix_spawn has completed, but before its return value has even been assigned to a variable?
Simulating this:
class FakeProcess():
def __init__(self):
# We create the process here at the OS-level,
# but just after, the user presses CTRL+C
raise KeyboardInterrupt()
def __enter__(self):
return self
def __exit__(self, _, __, ___):
# Never gets here
print("Got to exit")
p = None
try:
with FakeProcess() as p:
pass
finally:
print('p:', p)
This prints p: None, and does not print Got to exit.
This does suggest to me that a KeyboardInterrupt can prevent the cleanup of a process?
From https://docs.python.org/3/library/signal.html#note-on-signal-handlers-and-exception a KeyboardInterrupt is caused by Python's default SIGINT handler. And specifically it suggests when this can be raised:
If a signal handler raises an exception, the exception will be propagated to the main thread and may be raised after any bytecode instruction. Most notably, a KeyboardInterrupt may appear at any point during execution.
So it can be raised between, but not during, any Python bytecode instruction. So it all boils down to is "calling a function" (os.posix_spawn in this case) and "assigning its result to a variable" one instruction, or multiple.
And it's multiple. From https://pl.python.org/docs/lib/bytecodes.html there are STORE_* instructions which are all separate from calling a function.
Which means that in this case, a process can be made at the OS level that Python doesn’t know about.
https://docs.python.org/3/library/signal.html#note-on-signal-handlers-and-exception also states
Most Python code, including the standard library, cannot be made robust against this, and so a KeyboardInterrupt (or any other exception resulting from a signal handler) may on rare occasions put the program in an unexpected state.
Which hints as to how, I think pervasive, albeit maybe rare in practice, such issues can be in Python.
But https://docs.python.org/3/library/signal.html#note-on-signal-handlers-and-exception also gives a way of avoiding this:
applications that are complex or require high reliability should avoid raising exceptions from signal handlers. They should also avoid catching KeyboardInterrupt as a means of gracefully shutting down. Instead, they should install their own SIGINT handler.
Which you can do if you really need/want to. Taking the answer from https://stackoverflow.com/a/76919499/1319998, which is itself based on https://stackoverflow.com/a/71330357/1319998 you can essentially defer SIGINT/the KeyboardInterrupt
import signal
from contextlib import contextmanager
@contextmanager
def defer_signal(signum):
# Based on https://stackoverflow.com/a/71330357/1319998
original_handler = None
defer_handle_args = None
def defer_handle(*args):
nonlocal defer_handle_args
defer_handle_args = args
# Do nothing if
# - we don't have a registered handler in Python to defer
# - or the handler is not callable, so either SIG_DFL where the system
# takes some default action, or SIG_IGN to ignore the signal
# - or we're not in the main thread that doesn't get signals anyway
original_handler = signal.getsignal(signum)
if (
original_handler is None
or not callable(original_handler)
or threading.current_thread() is not threading.main_thread()
):
yield
return
try:
signal.signal(signum, defer_handle)
yield
finally:
# Note: if you have installed a signal handler for another
# signal that raises an exception, the original handler won't
# ever get re-attached
signal.signal(signum, original_handler)
if defer_handle_args is not None:
original_handler(*defer_handle_args)
to create a context manager with a stronger guarantee that you don't get a sort of zombie process due to a SIGINT during its creation:
@contextmanager
def PopenDeferringSIGINTDuringConstruction(*args, **kwargs):
# Very much like Popen, but defers SIGINT during its __init__, which is when
# the process starts at the OS level. This avoids what is essentially a
# zombie process - the process running but Python having no knowledge of it
#
# It doesn't guarentee that p will make it to client code, but should
# guarentee that the subprocesses __init__ method is not interrupted by a
# KeyboardInterrupt. And if __init__ raises an exception, then its
# __del__ method also shouldn't get interrupted, and so will clean
# up as best as it can
with defer_signal(signal.SIGINT):
p = Popen(*args, **kwargs)
with p:
yield p
which can be used as:
with PopenDeferringSIGINTDuringConstruction(['cat']) as p:
pass
Or... instead of dealing with signal handlers, as https://stackoverflow.com/a/842567/1319998 suggests, you can start the process in a thread, which doesn't get interrupted by signals. But then you are left with having to deal with both inter thread and iter process communication.
import threading
from contextlib import contextmanager
from subprocess import Popen
@contextmanager
def PopenInThread(*args, **kwargs):
# Very much like Popen, but starts it in a thread so its __init__
# (and maybe __del__ method if __init__ raises an exception) don't
# get interrupted by SIGINT/KeyboardInterrupt
p = None
exception = None
def target():
nonlocal p, exception
try:
p = Popen(*args, **kwargs)
except Exception as e:
exception = e
t = threading.Thread(target=target)
t.start()
t.join()
if exception is not None:
raise exception from None
with p:
yield p
used as
with PopenInThread(['cat']) as p:
pass
Choose your poison...