My team develops a GTK-Python application that dynamically spawns threads to handle longer running tasks. Recently, we added a feature for requesting some batch computation on demand and this has made the GUI unresponsive. My research and analysis shows that the problem seems to be contention for GIL that starves the GUI thread. The cheapest solution that we're considering is to move our custom threads to another process/GIL so that the threads spawned by the GUI framework run in one process but our dynamically spawned threads run in the other process.
I've been reading the multiprocessing docs and nothing jumps out at me that supports such a feature out of the box. Which options should I explore?
Here is one possibility. You create a child process thread_creator whose purpose is to create child threads. This is achieved by passing to thread_creator a multiprocessing.Queue instance that will contain tuples specifying the arguments for a new child thread to be created. A special sentinel value of None is a signal for the thread_creator process to terminate. When all non-daemon threads it has created terminate, then the child process will terminate and along with it any daemon threads that it has created.
In the code below we create a non-daemon thread sample_threaded_worker that will terminate after .5 seconds and a daemon thread another_threaded_worker that continuously loops printing a message every .1 seconds. After the thread_creator_process is asked to terminate, it will do so as soon as sample_threaded_worker completes. When that occurs, another_threaded_worker will automatically be terminated. Thus another_threaded_worker will be executing for approximately .5 seconds and will print out approximately 5 messages:
from multiprocessing import Process, Queue
from threading import Thread
import time
def sample_threaded_worker(x=1, y=2, debug=False):
time.sleep(.5)
if debug:
print('sample_threaded_worker', x, y)
def another_threaded_worker():
while True:
print('another_threaded_worker', time.time())
time.sleep(.1)
def thread_creator(queue):
while True:
request = queue.get()
if request is None:
break
# Unpack
target, args, kwargs, daemon = request
Thread(target=target, args=args, kwargs=kwargs, daemon=daemon).start()
def create_thread(queue, target=None, args=(), kwargs={}, daemon=None):
"""Helper function to create a child thread in the child process."""
queue.put((target, args, kwargs, daemon))
def main():
queue = Queue()
p = Process(target=thread_creator, args=(queue,))
p.start()
create_thread(queue, target=sample_threaded_worker, args=(5, 9), kwargs={'debug': True})
create_thread(queue, target=another_threaded_worker, daemon=True)
# Terminate process. All daemon threads that have been created will be killed
# when all non-daemon threads have completed.
queue.put(None)
p.join()
if __name__ == '__main__':
main()
Prints:
another_threaded_worker 1718880689.7697933
another_threaded_worker 1718880689.87115
another_threaded_worker 1718880689.9717884
another_threaded_worker 1718880690.0727093
another_threaded_worker 1718880690.1730008
sample_threaded_worker 5 9