I have the following situation:
There is a main.py with
def main():
...
schedule.schedule()
...
if __name__== "__main__":
main()
Then, I also have a file schedule.py with
def schedule()
...
toolbox = base.Toolbox()
creator.create("FitnessMin", base.Fitness, weights=(-1.0,))
creator.create("Individual", list, fitness=creator.FitnessMin)
toolbox.register('individual', init_indiv, creator.Individual, bounds=bounds)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)
toolbox.register("evaluate", fitness, data=args)
toolbox.register("mate", tools.cxTwoPoint)
toolbox.register("mutate", tools.mutFlipBit, indpb=0.05)
toolbox.register("select", tools.selTournament, tournsize=3)
# Further parameters
cxpb = 0.7
mutpb = 0.2
# Measure how long it takes to caluclate 1 generation
MAX_HOURS_GA = parameter._MAX_HOURS_GA
POPSIZE_GA = parameter._POPSIZE_GA
pool = multiprocessing.Pool(processes=4)
toolbox.register("map", pool.map)
pop = toolbox.population(n=POPSIZE_GA * len(bounds))
result = algorithms.eaSimple(pop, toolbox, cxpb, mutpb, 1, verbose=False)
Now, executing this gives me the following error:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
File "C:\Users\...\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "C:\Users\...\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\...\lib\multiprocessing\pool.py", line 110, in worker
task = get()
File "C:\Users\...\lib\multiprocessing\queues.py", line 354, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from 'C:\\Users...
Now, I do note that the DEAP documentation (https://deap.readthedocs.io/en/master/tutorials/basic/part4.html) says
Warning As stated in the multiprocessing guidelines, under Windows, a process pool must be protected in a >if __name__ == "__main__" section because of the way processes are initialized.
but that doesn't really help me as I certainly don't want to have all the toolbox.register(...)
in my main and it even might not be possible to do so. Just moving the creation of the pool
pool = multiprocessing.Pool(processes=4)
toolbox.register("map", pool.map)
to the main did not help.
There seem to be other people with similar issues, even fairly recently (https://github.com/rsteca/sklearn-deap/issues/59). For most of them, some sort of workaround seems to exist but none of them seems to fit in my situation or at least I couldn't figure out how to make them work. I've also tried moving around the order of registering the functions and initializing the pool, but with no luck. I've also tried using SCOOP instead but with similar results.
Any ideas?
The solution is to create "FitnessMin" and "Individual" in the global scope, i.e. in main.py:
import ...
creator.create("FitnessMin", base.Fitness, weights=(-1.0,))
creator.create("Individual", list, fitness=creator.FitnessMin)
def main():
...
schedule.schedule()
...
if __name__== "__main__":
main()