pythonmultiprocessing

Using multiprocessing pool in Python


Can someone point out what is wrong with this code snippet. It does not give any results.

import multiprocessing
    
results = []

def log_results(result):
    results.append(result)
    
def multiply(x, y):
    print(f"Gets here for process name {multiprocessing.current_process().name()}")
    return x * y
        
if __name__ == "__main__":
    pool = multiprocessing.Pool()
    numbers = [(1,1), (2,2), (3,3)]
    for x, y in numbers:
        print (f"Checking x {x} and y {y}")
        pool.apply_async(multiply, (x, y), callback=log_results)
    pool.close()
    pool.join()
    print(results)

results is an empty list which should not be in this case, right? I have used apply_async and map_async. Neither gives the right output. Can someone please help me here?


Solution

  • Edit: You made an edit to your code so now my answer below is out of date. The only two things I think need doing are:

    1. add an error_callback because I still think you need to ensure that the pool as written does not fail silently by default.
    2. rewrite multiprocessing.current_process().name() as multiprocessing.current_process().name.

    So:

    import multiprocessing
    
    results = []
    def log_results(result):
        results.append(result)
    
    def log_e(e):
      print(e)
    
    def multiply(x, y):
        print(f"Gets here for process name {multiprocessing.current_process().name}")
        return x * y
    
    
    pool = multiprocessing.Pool()
    numbers = [(1,1), (2,2), (3,3)]
    for x, y in numbers:
        print (f"Checking x {x} and y {y}")
        pool.apply_async(multiply, (x, y), callback=log_results,
                         error_callback=log_e)
    pool.close()
    pool.join()
    print(results)
    

    Old answer

    This drove me crazy for a moment but then it made sense.

    If I run it with multiply changed like this:

    def multiply(nums):
        print("print")
        return nums[0] * nums[1]
    

    It runs fine. You said in the comments "I do not think the function multiply is called in the first place." This is because there is a callback specified but no error_callback specified. The result of omitting an error callback is that your script is failing silently.

    You could check this with:

    import multiprocessing
    
    results = []
    def log_results(result):
        print(result)
    
    def log_e(e):
      print(e)
    
    def multiply(x, y):
        print(f"Gets here for process name {multiprocessing.current_process().name()}")
        return x * y
    
    pool = multiprocessing.Pool()
    numbers = [[1,1], [2,2], [3,3]]
    mapResult = pool.map_async(multiply, numbers, callback=log_results,
                               error_callback=log_e)
    
    pool.close()
    pool.join()
    

    Which gives:

    multiply() missing 1 required positional argument: 'y'
    

    And with multiply like so:

    def multiply(nums):
        return nums[0] * nums[1]
    

    It then returns [1, 4, 9]

    PS I am running Python 3.6.7