pythonmultithreadingjupyter-notebookmultiprocessingpython-multiprocessing

Multiprocessing example giving AttributeError in Jupyter Notebook


I am trying to implement multiprocessing in my code, and so, I thought that I would start my learning with some examples. I used the first example found in this documentation.

from multiprocessing import Pool
def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

When I run the above code I get an AttributeError: can't get attribute 'f' on <module '__main__' (built-in)>. I do not know why I am getting this error. I am also using Python 3.5 if that helps.


Solution

  • This problem seems to be a design feature of multiprocessing.Pool. See https://bugs.python.org/issue25053. For some reason Pool does not always work with objects not defined in an imported module. So you have to write your function into a different file and import the module.

    File: defs.py

    def f(x):
        return x*x
    

    File: run.py

    from multiprocessing import Pool
    import defs
    
     if __name__ == '__main__':
        with Pool(5) as p:
            print(p.map(defs.f, [1, 2, 3]))
    

    If you use print or a different built-in function, the example should work. If this is not a bug (according to the link), the given example is chosen badly.