pythonmultithreadingthreadpoolexecutorconcurrent.futures

processsing each element of dictionary of lists using concurrent.futures.ThreadPoolExecutor


suppose i have a dictionary such as

dict = {A:[1,2,3], B:[4,5,6], C:[7,8,9], ......}

I want to process each element of a particular key's list one after other but the individual keys could be processed in parallel using concurrent.futures.ThreadPoolExecutor

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
  ex.submit(process_element, contents of A)
  ex.submit(process_element, contents of B)
  ex.submit(process_element, contents of C)
  .
  .
  .

So the output should be

result of process_element A[0]
result of process_element B[0]
result of process_element C[0]

.
.
.

result of process_element A[1]

but not necessary in that order

what is wrong with the above method ?


Solution

  • It doesn't look like you're passing the list, but instead a variable named 'A'. I'm not sure when you run your code why your not getting an error. What you want is below

    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
      ex.submit(process_element, dict['A'])
      ex.submit(process_element, dict['B'])
      ex.submit(process_element, dict['C'])
    

    To process each element one by one, then you could simply use a list construction:

    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as ex:
      procsA = [ex.submit(process_element, elem) for elem in dict['A']
      ...
      ...