pythonparallel-processingmultiprocessingstarmap

Terminate all processes when condition is met


I am using starmap to run testing function.
Whats the best/safest way to terminate all processes when a process first finds the permutation [5,2,4,3,1]?

import multiprocessing as mp
import time

def testing(lts):
    # code ....
    start_time = time.time()
    for x in range(1,500000):
        gg = [1,2,3,4,5]
        random.shuffle(gg)
        ### if gg==[5,2,4,3,1] terminate all processes
    total_time = time.time() - start_time
    return total_time

if __name__ == '__main__':

    with mp.Pool(processes=4) as pool:
    ret = pool.starmap(testing, [(lst,) for x in range(4)])

Solution

  • I'm not very familiar with all this multiprocessing stuff, but setting a global variable only works with threads. multiprocessing creates different processes, so the global variable will be duplicated and a change of value is only visible in the current process.

    Instead, use a Manager, which creates a shared object between processes (dictionary, list for instance).

    There may be simpler ways, but I chose to create a shared Manager.list() object, which is initially empty.

    When the combination is found, just append something to the list. Test the list for emptyness in all processes. I've adapted your example so it works as-is (no need for numpy here, random.shuffle works fine.

    import multiprocessing as mp, random
    import time
    
    def testing(lst):
        # code ....
    
        start_time = time.time()
        for x in range(1,500000):
            gg = [1,2,3,4,5]
            random.shuffle(gg)
    
            if gg==[5,2,4,3,1]:
                print(gg)
                lst.append(1)  # reflected on all processes
            if lst:
                # list is no longer empty: stop
                print("stop")
                break
    
        total_time = time.time() - start_time
        return total_time
    
    if __name__ == '__main__':
        manager = mp.Manager()
        lst = manager.list()
    
        with mp.Pool(processes=4) as pool:
           ret = pool.starmap(testing, [(lst,) for x in range(4)])
    
        print(ret)
    

    a trace of execution:

    [5, 2, 4, 3, 1]
    stop
    stop
    stop
    stop
    [0.031249523162841797, 0.015624523162841797, 0.015624523162841797, 0.015624523162841797]
    

    as we see, the 4 processes have stopped, when one found the "solution".