I have a loop which does a bunch of CPU intensive calculations and appends the result to a list, per iteration.
How can I have that working in parallel. On C# there are concurrent containers, how does it work in ipyparallel?
From the ipyparallel documentation:
Python’s builtin map() functions allows a function to be applied to a sequence element-by-element. This type of code is typically trivial to parallelize.
http://ipyparallel.readthedocs.io/en/latest/direct.html#parallel-map
So it's a matter of using the map function to have it running on parallel, but how can I append the results to a list? Are there concurrent containers here?
So, what I have now is something like this:
results = []
for element in list:
outcome = very_heavy_computation_function(element)
results.append(outcome)
How can I do this in parallel?
You can achieve it in the following way.
Function foo represents calculations in the loop, parameters represents the data you want to iterate over if any. Even though foo sleeps for 10s the whole loop takes exactly 10s instead of 40s because I have 4 engines in my environment and functions run parallelly on the engines. A LoadBalancedView provides dynamic load balancing to evenly distribute work among engines.
from ipyparallel import Client
rc = Client()
view = rc.load_balanced_view()
def foo(param):
import time
time.sleep(10)
return param*2
parameters = list(range(4))
async_results = []
for p in parameters:
async_result = view.apply_async(foo, p)
async_results.append(async_result)
rc.wait_interactive(async_results)
results = [ar.get() for ar in async_results]
print(results)
Output:
4/4 tasks finished after 10 s
done
[0, 2, 4, 6]