How can I share a nested object between Python processes with write access for tasklets(coroutines)?
Here is a simplified example with an analogy just I wrote for asking this question properly;
First of all please install greenlet
package with: sudo pip install greenlet
In the example below:
Nature
class referenced by habitat
variableNature
class has an instance variable called animals
Nature
class, 8 different instances of Animal
class created and appended to animals
instance variable. Now if I'm correct this instance of Nature
is a nested object.live
instance functions of Animal
instances randomly switching using greenlet
package's switch()
function until global_counter
reaches to 1000. This live
function randomly changing the value of limbs
instance variable of Animal
instances.greentest.py:
import random
from greenlet import greenlet
global_counter = 0
class Animal():
def __init__(self,nature):
self.limbs = 0
nature.animals.append(self)
self.tasklet = greenlet(self.live)
def live(self,nature):
global global_counter
while True:
self.limbs = random.randint(1, 10)
global_counter += 1
if global_counter > 1000:
break
random.sample(nature.animals,1)[0].tasklet.switch(nature)
class Nature():
def __init__(self,how_many):
self.animals = []
for i in range(how_many):
Animal(self)
print str(how_many) + " animals created."
self.animals[0].live(self)
The result is:
>>> import greentest
>>> habitat = greentest.Nature(8)
8 animals created.
>>> habitat.animals[0].limbs
3
>>> greentest.global_counter
1002
Working as expected. Changing the value of limbs
and global_counter
(non-zero)
But when I add multiprocessing
to the equation;
greentest2.py:
import random
import multiprocessing
from greenlet import greenlet
global_counter = 0
class Animal():
def __init__(self,nature):
self.limbs = 0
nature.animals.append(self)
self.tasklet = greenlet(self.live)
def live(self,nature):
global global_counter
while True:
self.limbs = random.randint(1, 10)
global_counter += 1
if global_counter > 1000:
break
random.sample(nature.animals,1)[0].tasklet.switch(nature)
class Nature():
def __init__(self,how_many):
self.animals = []
for i in range(how_many):
Animal(self)
print str(how_many) + " animals created."
#self.animals[0].live(self)
jobs = []
for i in range(2):
p = multiprocessing.Process(target=self.animals[0].live, args=(self,))
jobs.append(p)
p.start()
The result is not as expected:
>>> import greentest2
>>> habitat = greentest2.Nature(8)
8 animals created.
>>> habitat.animals[0].limbs
0
>>> greentest2.global_counter
0
Both the values of limbs
and global_counter
is unchanged (zero). I think this is because instances of Animal
class and global_counter
is not shared between processes. So how can I share this instance of Nature
class or these instances of Animal
class between processes?
ADDITION according to @noxdafox 's answer;
greentest3.py:
import random
import multiprocessing
from greenlet import greenlet
global_counter = multiprocessing.Value('i', 0)
class Animal():
def __init__(self,nature):
self.limbs = 0
nature.animals.append(self)
self.tasklet = greenlet(self.live)
def live(self,nature):
global global_counter
while True:
self.limbs = random.randint(1, 10)
global_counter.value += 1
if global_counter.value > 1000:
break
random.sample(nature.animals,1)[0].tasklet.switch(nature)
class Nature():
def __init__(self,how_many):
self.animals = []
for i in range(how_many):
Animal(self)
print str(how_many) + " animals created."
#self.animals[0].live(self)
jobs = []
for i in range(2):
p = multiprocessing.Process(target=self.animals[0].live, args=(self,))
jobs.append(p)
p.start()
and then result is:
>>> import greentest3
>>> habitat = greentest3.Nature(8)
8 animals created.
>>> habitat.animals[0].limbs
0
>>> greentest3.global_counter.value
1004
I was perfectly aware that global_counter
can be shared with this method since it's an integer but I'm actually asking how to share the instances of Nature
and Animal
classes between processes.
Different processes do not share their memory.
If what you need to share is a single variable, you probably can use multiprocessing.Value
import multiprocessing
def function(counter):
counter.value += 1
counter = multiprocessing.Value('i')
p = multiprocessing.Process(target=function, args=(counter))
p.start()
p.join()
EDIT: answering according to updates.
There is no abstraction mechanism allowing to share entire objects in memory. Shared memory is usually implemented as a simple array where processes can read/write once acquired the resource.
Moreover, OOP and threading/multiprocessing don't mix well together. IMHO should be considered an anti-pattern. On top of complex objects, you add the concurrent access and modification of their properties. This is a one way ticket for long and tedious debugging sessions.
The recommended pattern is the use of message queues. Imagining Threads and Processes as isolated entities which communicates via specific channels significantly simplifies the problem.