pythonconcurrent.futures

How can I keep track of which function returned which value, with concurrent.futures.ThreadPoolExecutor()?


My code is as follows:

with concurrent.futures.ThreadPoolExecutor() as executor:
    length = len(self.seq)
    futures = [None] * length
    results = []
    for i in range(length):
        f,args,kwargs = self.seq[i]
        future = executor.submit(f, *args, **kwargs)
        futures[i] = future

    for f in concurrent.futures.as_completed(starts):
        results.append(f.result())

I'd like the return values in results to be ordered by the order of the functions they were returned from in self.seq, but my code orders them in the order of whichever returns first. How can I do this?

For clarification, .as_completed() is not what I'm looking for. I want to maintain the order of the original function list.


Solution

  • The simplest answer is to pass a counter argument along with the rest of your args and kwargs, where you set counter to be i. You then need to modify f to return the counter along with whatever other values its supposed to return. Your results can then be sorted by the counter.

    Alternatively, create a results array of the right size, and as each result comes back, the counter value tells you where to put it in the array.


    If you do not have access to f, then just write a wrapper around it:

    def f_wrapper(*args, *, counter, **kwargs):
        result = f(*args, **kwargs)
        return counter, result
    

    Likewise, you'll write

    future = executor.submit(f_wrapper, *args, **kwargs | dict(counter=i))