pythonmultiprocessingprofilerpathos

cProfiler working weirdly with multiprocessing


I got an error for this code:

from pathos.multiprocessing import ProcessingPool
def diePlz(im):
    print('Whoopdepoop!')   
def caller():
    im = 1
    pool = ProcessingPool()
    pool.map(diePlz,[im,im,im,im]) 

if __name__=='__main__':
    caller()    

when I ran it with the cProfiler: (python3 -m cProfile testProfiler.py)

multiprocess.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/rohit/.local/lib/python3.6/site-packages/pathos/helpers/mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "testProfiler.py", line 3, in diePlz
    print('Whoopdepoop!')
NameError: name 'print' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/cProfile.py", line 160, in <module>
    main()
  File "/usr/lib/python3.6/cProfile.py", line 153, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/usr/lib/python3.6/cProfile.py", line 20, in runctx
    filename, sort)
  File "/usr/lib/python3.6/profile.py", line 64, in runctx
    prof.runctx(statement, globals, locals)
  File "/usr/lib/python3.6/cProfile.py", line 100, in runctx
    exec(cmd, globals, locals)
  File "testProfiler.py", line 11, in <module>
    caller()    
  File "testProfiler.py", line 8, in caller
    pool.map(diePlz,[im,im,im,im]) 
  File "/home/rohit/.local/lib/python3.6/site-packages/pathos/multiprocessing.py", line 137, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/rohit/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 644, in get
    raise self._value
NameError: name 'print' is not defined

But when I ran it without the cProfiler:

$ python3 testProfiler.py 
Whoopdepoop!
Whoopdepoop!
Whoopdepoop!
Whoopdepoop!

The code that I've provided is a minimal working example for the problem. There is a much larger code that I want to debug, but am not able to do so because cProfiler keeps raising weird errors.

In this case, the point of importance is

NameError: name 'print' is not defined

which means python3 is not able to recognize print itself. In my code, it was not able to recognize range.


Solution

  • So, I realize this is a long time after the original post, but I have this exact same issue.

    In my case I was getting the exact same error as the original post - python builtin functions such as print() or len() resulted in errors like this:

    NameError: name 'len' is not defined
    

    I'm currently running multiprocess version 0.70.11.1 and dill version 0.3.3 (components of pathos that make process based parallelism work).

    Based on what I found in an issue comment: https://github.com/uqfoundation/pathos/issues/129#issuecomment-536081859 one of the package authors recommends trying:

    import dill
    dill.settings['recurse'] = True
    

    At least in my case, the above fixed the error!