I am learning parallel computation in ipython. I came across an example,
from ipyparallel import Client
rc = Client()
rc.block = True
print(rc.ids)
def mul(a,b):
return a*b
dview = rc[:]
print(dview.apply(mul, 5, 6))
print(rc[0].apply(mul, 5, 5))
print(rc[1].apply(mul, 5, 9))
In the above code, when dview.apply is called, it passes the same set of arguments for all the clients. I learned to call each clients separately. But if the clients are to do data intensive tasks, is there a way to pass different arguments through dview.apply since that is rather the point of doing parallel computation.
If there is no other way, can we make each client call asynchronous, so the tasks will be done parallel instead of waiting for results from the first client when individual clients are called ?.
In general, parallel computations can be expressed as a maps, where you pass sequences of arguments:
dview = rc[:]
inputs = [6, 5, 9]
results = dview.map(mul, [5] * len(inputs), inputs)
can we make each client call asynchronous
yes, you can use view.apply_async to return a Future corresponding to the result:
ar = view.apply_async(mul, 5, 6)
result = ar.get()