pythonpython-3.xwindowsmultiprocessingdeap

Python DEAP and multiprocessing on Windows: AttributeError


I have the following situation:

There is a main.py with

def main():
    ...
    schedule.schedule()
    ...

if __name__== "__main__":
    main()

Then, I also have a file schedule.py with

def schedule()
   ...
    toolbox = base.Toolbox()

    creator.create("FitnessMin", base.Fitness, weights=(-1.0,))
    creator.create("Individual", list, fitness=creator.FitnessMin)

    toolbox.register('individual', init_indiv, creator.Individual, bounds=bounds)
    toolbox.register("population", tools.initRepeat, list, toolbox.individual)

    toolbox.register("evaluate", fitness, data=args)
    toolbox.register("mate", tools.cxTwoPoint)
    toolbox.register("mutate", tools.mutFlipBit, indpb=0.05)
    toolbox.register("select", tools.selTournament, tournsize=3)

    # Further parameters
    cxpb = 0.7
    mutpb = 0.2

    # Measure how long it takes to caluclate 1 generation
    MAX_HOURS_GA = parameter._MAX_HOURS_GA
    POPSIZE_GA = parameter._POPSIZE_GA
    pool = multiprocessing.Pool(processes=4)
    toolbox.register("map", pool.map)
    pop = toolbox.population(n=POPSIZE_GA * len(bounds))
    result = algorithms.eaSimple(pop, toolbox, cxpb, mutpb, 1, verbose=False)

Now, executing this gives me the following error:

Process SpawnPoolWorker-1:
Traceback (most recent call last):
  File "C:\Users\...\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\...\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\...\lib\multiprocessing\pool.py", line 110, in worker
    task = get()
  File "C:\Users\...\lib\multiprocessing\queues.py", line 354, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from 'C:\\Users...

Now, I do note that the DEAP documentation (https://deap.readthedocs.io/en/master/tutorials/basic/part4.html) says

Warning As stated in the multiprocessing guidelines, under Windows, a process pool must be protected in a >if __name__ == "__main__" section because of the way processes are initialized.

but that doesn't really help me as I certainly don't want to have all the toolbox.register(...) in my main and it even might not be possible to do so. Just moving the creation of the pool

    pool = multiprocessing.Pool(processes=4)
    toolbox.register("map", pool.map)

to the main did not help.

There seem to be other people with similar issues, even fairly recently (https://github.com/rsteca/sklearn-deap/issues/59). For most of them, some sort of workaround seems to exist but none of them seems to fit in my situation or at least I couldn't figure out how to make them work. I've also tried moving around the order of registering the functions and initializing the pool, but with no luck. I've also tried using SCOOP instead but with similar results.

Any ideas?


Solution

  • The solution is to create "FitnessMin" and "Individual" in the global scope, i.e. in main.py:

    import ...
    
    creator.create("FitnessMin", base.Fitness, weights=(-1.0,))
    creator.create("Individual", list, fitness=creator.FitnessMin)
    
    def main():
        ...
        schedule.schedule()
        ...
    
    if __name__== "__main__":
        main()