pythonpython-3.xparallel-processingscipydifferential-evolution

python scipy differential evolution optimization fails with workers not 1


I am trying to use the workers parameter of the scipy differential evolution algo.

When I put it as 1, my script runs with no issues. If I put something different, it fails with the following traceback :

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/_lib/_util.py", line 419, in __call__
    return self._mapfunc(func, iterable)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 878, in _calculate_population_energies
    parameters_pop[0:nfevs]))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/_lib/_util.py", line 422, in __call__
    raise TypeError("The map-like callable must be of the"
TypeError: The map-like callable must be of the form f(func, iterable)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main_parallel.py", line 323, in <module>
    optimizer(args.asset_class, udl, tenor, args.criteria, params_decay, params, couch_server, args.num_workers)
  File "main_parallel.py", line 269, in optimizer
    maxiter=5, workers=num_workers, mutation=(0.5, 1.5), recombination=0.8)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 306, in differential_evolution
    ret = solver.solve()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 745, in solve
    self.population[self.feasible]))
  File "/home/ubuntu/.local/lib/python3.6/site-packages/scipy/optimize/_differentialevolution.py", line 883, in _calculate_population_energies
    raise RuntimeError("The map-like callable must be of the"
RuntimeError: The map-like callable must be of the form f(func, iterable), returning a sequence of numbers the same length as 'iterable'

Could someone please help in explaining how to use this parameter (specific objective function ? other constraints ?) ?

Also an example of a simple python differential evolution optimization using multiple cores would be appreciated.


Solution

  • The critical piece of the differential evolution documentation regarding the usage of workers is the caveat that when worker != 1, it

    [...] Requires that func be pickleable

    There are several hints in the errors raised which indicate the func is not pickleable, namely

        self._mapfunc(func, iterable)
        ...
        self._send_bytes(_ForkingPickler.dumps(obj))
        ...
    TypeError: cannot serialize '_io.TextIOWrapper' object
    

    Clearly, there was an attempt at pickling func made which failed due, presumably, to func not being pickleable. It looks like workers is attempted to be interpreted as a map-like or callable, as the other signature in the documentation indicates. Predictably this fails as well since workers is in fact an int.

    The documentation contains a complete example where workers != 1 which works correctly,

    from scipy.optimize import differential_evolution, rosen
    bounds = [(0,2), (0, 2), (0, 2), (0, 2), (0, 2)]
    result = differential_evolution(rosen, bounds, updating='deferred',
                                    workers=-1)
    result.x, result.fun
    

    If you refactor the func being used in scipy.optimize.differential_evolution to be serializable via pickling you should be able to use workers != 1 successfully.