pythonparallel-processingipythonipython-parallel

IPython parallel DirectView.apply does not run in parallel


For some reason I cannot get IPython parallel function DirectView.apply() to really parallelize a function call. DirectView.map() works as expected. Am I doing something wrong here? A working example script

import time
from datetime import datetime
from ipyparallel import Client, require

@require(time)
def wait(seconds=1):
    time.sleep(seconds)

if __name__=='__main__':
    client = Client()
    print('engine ids: {}'.format(client.ids))
    dview = client.direct_view((0, 1, 2, 3))
    dview.block = False
    print('view targets: {}'.format(dview.targets))

    print('dview.apply...')
    t0 = datetime.now()
    results = [dview.apply(wait) for i in range(4)]
    while len(results) > 0:
        results.pop(0).get()
    print('time: {}'.format(datetime.now() - t0))

    print('dview.map... ')
    t0 = datetime.now()
    results = dview.map(wait, [1]*4)
    print('time: {}'.format(datetime.now() - t0))

prints

engine ids: [0, 1, 2, 3]
view targets: [0, 1, 2, 3]
dview.apply...
time: 0:00:04.021680
dview.map... 
time: 0:00:01.013941

showing that apply clearly does not perform as I expected.

My system is Ubuntu 14.04, Python 3.4.3, IPython 4.2.0.


Solution

  • Seems that I misinterpreted the documentation. Here it says that apply(f, *args, **kwargs) calls f(*args, **kwargs) on remote engines, returning the result.

    If I understand correctly, this means that the function is applied on all of the engines, instead of running it once on just one of them.