pythonmultithreadingdaemonconcurrent.futures

The workers in ThreadPoolExecutor is not really daemon


The thing I cannot figure out is that although ThreadPoolExecutor uses daemon workers, they will still run even if main thread exit.

I can provide a minimal example in python3.6.4:

import concurrent.futures
import time


def fn():
    while True:
        time.sleep(5)
        print("Hello")


thread_pool = concurrent.futures.ThreadPoolExecutor()
thread_pool.submit(fn)
while True:
    time.sleep(1)
    print("Wow")

Both main thread and the worker thread are infinite loops. So if I use KeyboardInterrupt to terminate main thread, I expect that the whole program will terminate too. But actually the worker thread is still running even though it is a daemon thread.

The source code of ThreadPoolExecutor confirms that worker threads are daemon thread:

t = threading.Thread(target=_worker,
                     args=(weakref.ref(self, weakref_cb),
                           self._work_queue))
t.daemon = True
t.start()
self._threads.add(t)

Further, if I manually create a daemon thread, it works like a charm:

from threading import Thread
import time


def fn():
    while True:
        time.sleep(5)
        print("Hello")


thread = Thread(target=fn)
thread.daemon = True
thread.start()
while True:
    time.sleep(1)
    print("Wow")

So I really cannot figure out this strange behavior.


Solution

  • Suddenly... I found why. According to much more source code of ThreadPoolExecutor:

    # Workers are created as daemon threads. This is done to allow the interpreter
    # to exit when there are still idle threads in a ThreadPoolExecutor's thread
    # pool (i.e. shutdown() was not called). However, allowing workers to die with
    # the interpreter has two undesirable properties:
    #   - The workers would still be running during interpreter shutdown,
    #     meaning that they would fail in unpredictable ways.
    #   - The workers could be killed while evaluating a work item, which could
    #     be bad if the callable being evaluated has external side-effects e.g.
    #     writing to a file.
    #
    # To work around this problem, an exit handler is installed which tells the
    # workers to exit when their work queues are empty and then waits until the
    # threads finish.
    
    _threads_queues = weakref.WeakKeyDictionary()
    _shutdown = False
    
    def _python_exit():
        global _shutdown
        _shutdown = True
        items = list(_threads_queues.items())
        for t, q in items:
            q.put(None)
        for t, q in items:
            t.join()
    
    atexit.register(_python_exit)
    

    There is an exit handler which will join all unfinished worker...