pythonclosuresstarmap

When using python's starmap, why is the location of a variable passed in using a closure different from that passed in using partial?


When I run the following code I see that variables z and w inside the function have different locations, but I don't understand why this is the case.

import multiprocessing
from functools import partial

class SomeClass:
    pass

w = SomeClass()
print("w outside function")
print(w)

def some_function(x, y, z):
    print("w inside function")
    print(w)

    print("z")
    print(z)

    return x * 2


    pool = multiprocessing.Pool(processes=4)
    some_function_partial = partial(some_function, z=w)
    summary = pool.starmap(some_function_partial, [(3, 2)])
    pool.close()
    pool.join()

Output:

w outside function
<__main__.SomeClass object at 0x104916080>
w inside function
<__main__.SomeClass object at 0x104916080>
z
<__main__.SomeClass object at 0x104ef66d8>

If the partial function is run on the main thread all three print statements show that the variables being printed refer to the same instance.

If it is run with starmap the location of z inside the function differs from w outside the function. This I would expect, as the function is being sent to a difference process to be run and presumably serialised/deserialised. However, when I run with starmap the location of w inside and outside the function is the same, and I feel that I'm missing some fundamental understanding as to why... Any help would be much appreciated!


Solution

  • In case anyone stumbles across this: I believe the reason is because when new process is created on this system, the existing heap is copied, which is why w is till effectively in scope. Feel free to correct me if this is wrong though!