I am trying to use the workers
parameter of the scipy differential evolution algo.
When I put it as 1, my script runs with no issues. If I put something different, it fails with the following traceback :
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/_lib/_util.py", line 419, in __call__
return self._mapfunc(func, iterable)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
put(task)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 878, in _calculate_population_energies
parameters_pop[0:nfevs]))
File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/_lib/_util.py", line 422, in __call__
raise TypeError("The map-like callable must be of the"
TypeError: The map-like callable must be of the form f(func, iterable)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main_parallel.py", line 323, in <module>
optimizer(args.asset_class, udl, tenor, args.criteria, params_decay, params, couch_server, args.num_workers)
File "main_parallel.py", line 269, in optimizer
maxiter=5, workers=num_workers, mutation=(0.5, 1.5), recombination=0.8)
File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 306, in differential_evolution
ret = solver.solve()
File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 745, in solve
self.population[self.feasible]))
File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 883, in _calculate_population_energies
raise RuntimeError("The map-like callable must be of the"
RuntimeError: The map-like callable must be of the form f(func, iterable), returning a sequence of numbers the same length as 'iterable'
Could someone please help in explaining how to use this parameter (specific objective function ? other constraints ?) ?
Also an example of a simple python differential evolution optimization using multiple cores would be appreciated.
The critical piece of the differential evolution documentation regarding the usage of workers is the caveat that when worker != 1
, it
[...] Requires that
func
be pickleable
There are several hints in the errors raised which indicate the func
is not pickleable, namely
self._mapfunc(func, iterable) ... self._send_bytes(_ForkingPickler.dumps(obj)) ... TypeError: cannot serialize '_io.TextIOWrapper' object
Clearly, there was an attempt at pickling func
made which failed due, presumably, to func
not being pickleable. It looks like workers
is attempted to be interpreted as a map-like or callable, as the other signature in the documentation indicates. Predictably this fails as well since workers
is in fact an int
.
The documentation contains a complete example where workers != 1
which works correctly,
from scipy.optimize import differential_evolution, rosen
bounds = [(0,2), (0, 2), (0, 2), (0, 2), (0, 2)]
result = differential_evolution(rosen, bounds, updating='deferred',
workers=-1)
result.x, result.fun
If you refactor the func
being used in scipy.optimize.differential_evolution
to be serializable via pickling you should be able to use workers != 1
successfully.