I get this strange behaviour (random TypeError exception is thrown) whenever I interrupt/kill a sleeping parent thread from a child thread in python 3 but the strange behaviour ONLY appears if the python script was initiated by a shell script with an & argument to make it run in the background.
So here's the minimum reproducible python code I could come up with that can trigger this issue.
User@MSI: ~/test $ cat m.py
import threading
import _thread
import time
def child_thread():
time.sleep(1)
print('child interrupting parent')
_thread.interrupt_main()
if __name__ == '__main__':
t = threading.Thread(target=child_thread, args=())
t.start()
print('parent looping')
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
print('caught interruption raised from user or child thread :)')
except TypeError as e:
print('why would I ever catch a TypeError?')
raise
except Exception as e:
print('Strange Exception Received ')
raise
It's a very simple script where a parent initiates a child thread and then loops forever and the child thread (after 1 second) interrupts the main thread which SHOULD raise a KeyboardInterruption towards the main thread.
Now if I invoke my script using a bash script with the background flag & at the end,
(the below output makes no sense)
User@MSI: ~/test $ cat ./m.sh
#!/bin/bash
python3 m.py &
User@MSI: ~/test $ ./m.sh
parent looping
child interrupting parent
why would I ever catch a TypeError?
Traceback (most recent call last):
File "m.py", line 17, in <module>
time.sleep(1)
TypeError: 'int' object is not callable
below I show the output of my script behaving completely normal when I run it from
And all three behave as expected... (the below output is as expected)
User@MSI: ~/test $ python3 m.py
parent looping
child interrupting parent
caught interruption raised from user or child thread :)
Same result when running it in the background with & (the below output is as expected)
User@MSI: ~/test $ python3 m.py &
[1] 5884
parent looping
child interrupting parent
caught interruption raised from user or child thread :)
[1]+ Done python3 m.py
And even from a one line script called m.sh that executes m.py (the below output is as expected)
User@MSI: ~/test $ cat m.sh
#!/bin/bash
python3 m.py
User@MSI: ~/test $ ./m.sh
parent looping
child interrupting parent
caught interruption raised from user or child thread :)
I'm completely dumbfounded and have no idea what time.sleep and TypeErrors have to with how I invoked my script and specifically invoking it from a shell script and from the backgrounded. This is one of the strangest errors I've encountered.
If it matters, I'm running Python 3.6.12 and Debian 9.12
I hope someone can figure this out.
EDIT: Here are comparisons of the bytecode for the buggy version's (run from shell script with &) output And here's the good version's (run from terminal) output
And for easy comparison, here's a diff of the bytecode. The only difference is the location of the child thread in memory.
This was a bug in CPython, issue23395.
It was fixed in Python 3.8 by PR #7778. The changelog mentions it at the last "Library" item for 3.8.0 beta 1, which says:
_thread.interrupt_main()now avoids setting the Python error status if theSIGINTsignal is ignored or not handled by Python.
The fix was also backported to Python 3.7 in PR #13541, so you should find it reproducible in Python 3.7.3 and then fixed in Python 3.7.4.
Thomas Kluyver explains why it's a TypeError:
This fails with an error message "TypeError: 'int' object is not callable", and a traceback completely disconnected from the cause of the error, presumably because it's not coming from the usual Python stack.
The problem appears to be that interrupt_main sets (in the C code) Handlers[SIGINT].tripped, which is only expected to occur when the handler is a Python function. When PyErr_CheckSignals() runs, it tries to call Handlers[SIGINT].func as a Python function, but it's a Python integer, causing the error.
You're correct that the TypeError traceback "makes no sense".
The reason it only happens when the Python script m.py was run from a backgrounded task in a bash script m.sh is because job control (ability to suspend/resume tasks) is disabled wen executing that way. If job control is disabled, Python will set signum 2 handler to signal.SIG_IGN, which ignores the interrupt. Otherwise, the signum 2 handler will be signal.default_int_handler, which raises a KeyboardInterrupt exception. This is not a CPython-specific thing, it's a POSIX thing (ref).
You can confirm this by adding set -m (i.e. turn on the "monitor" shell option) within the bash script:
#!/bin/bash
set -m
python3 m.py &
That should also avoid triggering the bug.