pythonpython-3.xmultiprocessingdaskdask-delayed

unpacking a dask delayed object of list of tuples


I have a function returning a tuple of two elements. The function is called with pool starmap to generate a list of tuples which are unpacked to two lists.

def func():
   #...some operations
   return (x,y)

def MP_a_func(func,iterable,proc,chunk):
    pool=multiprocessing.Pool(processes=proc)
    Result=pool.starmap(func,iterable,chunksize=chunk)
    pool.close()
    return Result
##
if __name__ == '__main__':
    results=MP_a_func(func,iterable,proc,chunk)

a,b=zip(*results)

I now wish to use dask delayed API as the following

if __name__ == '__main__':
    results=delayed(MP_a_func(func,iterable,proc,chunk))

is it possible to unpack tuples in the delayed object without using results.compute() ?

Thank your for your help


Solution

  • If you know the number of outputs, the delayed function (or decorator) takes an optional nout arguments, and this will split the single delayed into that many delayed outputs. This sounds like exactly what you need.