I have simple map-reduce type algorithm, which I want to implement in python and make use of multiple cores.
I read somewhere that threads using native thread module in 2.6 dont make use of multiple cores. is that true?
I even implemented it using stackless python however i am getting into weird errors [Update: a quick search showed that the stack less does not allows multiple cores So are their any other alternatives?]
def Propagate(start,end):
print "running Thread with range: ",start,end
def maxVote(nLabels):
count = {}
maxList = []
maxCount = 0
for nLabel in nLabels:
if nLabel in count:
count[nLabel] += 1
else:
count[nLabel] = 1
#Check if the count is max
if count[nLabel] > maxCount:
maxCount = count[nLabel];
maxList = [nLabel,]
elif count[nLabel]==maxCount:
maxList.append(nLabel)
return random.choice(maxList)
for num in range(start,end):
node=MapList[num]
nLabels = [Label[k] for k in Adj[node]]
if (nLabels!=[]):
Label[node] = maxVote(nLabels)
else:
Label[node]=node
However in above code the values assigned to Label, that is the change in dictionary are lost.
Above propagate function is used as callable for MicroThreads (i.e. TaskLets)
Use the multiprocessing module of the standard library -- it mimics the interface of the threading
module (to ease porting of existing multithreaded code) and can fully use all the cores you have. threading and stackless are both single-core (and indeed, stackless is non-preemptive, so programming for it is quite different than programming for either threading or multiprocessing).