pythonmultiprocessingcoroutinegreenletsobject-sharing

Sharing a nested object between Python processes with write access for tasklets(coroutines)?


How can I share a nested object between Python processes with write access for tasklets(coroutines)?

Here is a simplified example with an analogy just I wrote for asking this question properly;

First of all please install greenlet package with: sudo pip install greenlet

In the example below:

greentest.py:

import random
from greenlet import greenlet

global_counter = 0

class Animal():

    def __init__(self,nature):
        self.limbs = 0
        nature.animals.append(self)
        self.tasklet = greenlet(self.live)

    def live(self,nature):
        global global_counter
        while True:
            self.limbs = random.randint(1, 10)
            global_counter += 1
            if global_counter > 1000:
                break
            random.sample(nature.animals,1)[0].tasklet.switch(nature)

class Nature():

    def __init__(self,how_many):
        self.animals = []
        for i in range(how_many):
            Animal(self)
        print str(how_many) + " animals created."
        self.animals[0].live(self)

The result is:

>>> import greentest
>>> habitat = greentest.Nature(8)
8 animals created.
>>> habitat.animals[0].limbs
3
>>> greentest.global_counter
1002

Working as expected. Changing the value of limbs and global_counter (non-zero)

But when I add multiprocessing to the equation;

greentest2.py:

import random
import multiprocessing
from greenlet import greenlet

global_counter = 0

class Animal():

    def __init__(self,nature):
        self.limbs = 0
        nature.animals.append(self)
        self.tasklet = greenlet(self.live)

    def live(self,nature):
        global global_counter
        while True:
            self.limbs = random.randint(1, 10)
            global_counter += 1
            if global_counter > 1000:
                break
            random.sample(nature.animals,1)[0].tasklet.switch(nature)

class Nature():

    def __init__(self,how_many):
        self.animals = []
        for i in range(how_many):
            Animal(self)
        print str(how_many) + " animals created."
        #self.animals[0].live(self)
        jobs = []
        for i in range(2):
            p = multiprocessing.Process(target=self.animals[0].live, args=(self,))
            jobs.append(p)
            p.start()

The result is not as expected:

>>> import greentest2
>>> habitat = greentest2.Nature(8)
8 animals created.
>>> habitat.animals[0].limbs
0
>>> greentest2.global_counter
0

Both the values of limbs and global_counter is unchanged (zero). I think this is because instances of Animal class and global_counteris not shared between processes. So how can I share this instance of Nature class or these instances of Animal class between processes?

ADDITION according to @noxdafox 's answer;

greentest3.py:

import random
import multiprocessing
from greenlet import greenlet

global_counter = multiprocessing.Value('i', 0)

class Animal():

    def __init__(self,nature):
        self.limbs = 0
        nature.animals.append(self)
        self.tasklet = greenlet(self.live)

    def live(self,nature):
        global global_counter
        while True:
            self.limbs = random.randint(1, 10)
            global_counter.value += 1
            if global_counter.value > 1000:
                break
            random.sample(nature.animals,1)[0].tasklet.switch(nature)

class Nature():

    def __init__(self,how_many):
        self.animals = []
        for i in range(how_many):
            Animal(self)
        print str(how_many) + " animals created."
        #self.animals[0].live(self)
        jobs = []
        for i in range(2):
            p = multiprocessing.Process(target=self.animals[0].live, args=(self,))
            jobs.append(p)
            p.start()

and then result is:

>>> import greentest3
>>> habitat = greentest3.Nature(8)
8 animals created.
>>> habitat.animals[0].limbs
0
>>> greentest3.global_counter.value
1004

I was perfectly aware that global_counter can be shared with this method since it's an integer but I'm actually asking how to share the instances of Nature and Animal classes between processes.


Solution

  • Different processes do not share their memory.

    If what you need to share is a single variable, you probably can use multiprocessing.Value

    import multiprocessing
    
    def function(counter):
        counter.value += 1
    
    counter = multiprocessing.Value('i')
    p = multiprocessing.Process(target=function, args=(counter))
    p.start()
    p.join()
    

    EDIT: answering according to updates.

    There is no abstraction mechanism allowing to share entire objects in memory. Shared memory is usually implemented as a simple array where processes can read/write once acquired the resource.

    Moreover, OOP and threading/multiprocessing don't mix well together. IMHO should be considered an anti-pattern. On top of complex objects, you add the concurrent access and modification of their properties. This is a one way ticket for long and tedious debugging sessions.

    The recommended pattern is the use of message queues. Imagining Threads and Processes as isolated entities which communicates via specific channels significantly simplifies the problem.