python-3.xmathematical-optimizationevolutionary-algorithmpygmo

Get evolution log from a pygmo archipelago


The question is pretty straightforward and maybe stupid, but here we go:

as in here (https://esa.github.io/pagmo2/docs/python/algorithms/py_algorithms.html)
if you evolve a single population you can get the log of your algo.evolve() call as follows:

from pygmo import *
algo = algorithm(de1220(gen = 500))
algo.set_verbosity(100)
prob = problem(rosenbrock(10))
pop = population(prob, 20)
pop = algo.evolve(pop) 
uda = algo.extract(de1220)
uda.get_log() 
[(1, 20, 285652.7928977573, 0.551350234239449, 0.4415510963067054, 16, 43.97185788345982, 2023791.5123259544), ...

If you leverage the power of pygmo to parallelize evolution using an archipelago you would do something like:

archi = archipelago(n = 8, algo = algo, prob = rosenbrock(5), pop_size = 10, seed = 32)
archi.evolve()

However the archipelagos does not have an extract() method (as the algorithms do), nor a get_algorithm() one (as do islands), nor anything else obvious enough in the documentation (at least for me) that would do the job...

archi.extract(de1220)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'archipelago' object has no attribute 'extract'


archi.get_algorithm()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'archipelago' object has no attribute 'get_algorithm'

So, how do I get the output of algo.set_verbosity(100) into a file and not just on the stdout?
And, once here, is there a way to get it organized by island and not interleaved as it is printed on stdout?
(I get that while running every island prints the report when reaching the give trigger, but if everything is stored it should be possible to sort it)

Thanks!


Solution

  • Pretty badly documented, I wasted quite some time on it...
    I'm not sure this is the best/proper/faster way to do it, but works:

    1. Turns out (and is nowhere in the docs) that you can iterate through the islands of an archipelago with a simple for loop.
    2. Next, chaining the .get_algorithm() method of the islands and the .extract() method of the algorithm it is possible to extract the log of your run island by island. A bit of fun with numpy/pandas and it all gets in a decent understandable format.

    code-wise:

    # set up a dummy archipelago
    algo = algorithm(de1220(gen = 50))
    algo.set_verbosity(25)
    prob = problem(rosenbrock(10))
    archi = pg.archipelago(n=5,algo=algo, prob=prob, pop_size=10)
    
    # evolve the archipelago
    archi.evolve()
    archi.wait()
    
    # set up df
    tot_df = pd.DataFrame(columns = ["Gen", "F.evals.", "Best fit", "mutation", "crossing over", "Variant", "dx", "df", "island_#"])
    
    # here's the 'magic'
    for i, island in enumerate(archi): # iterate through islands
       a = island.get_algorithm()      # get algorithm from island
       uda = a.extract(de1220)         # extract algorithm from algorithm object
       log = uda.get_log()             # get the log. Comes as list of tuples
    
       # reshape log
       df = pd.DataFrame(np.asarray(log), columns = ["Gen", "F.evals.", "Best fit","mutation", "crossing over", "Variant", "dx", "df"])
       df["island_#"] = i              # add island ID
       tot_df = pd.concat([tot_df,df], axis='index', ignore_index=True) # merge with total df
    
    tot_df.head(10)
    
       Gen  F.evals.       Best fit  mutation  crossing over  Variant         dx  \
    0   1.0      10.0  345333.467771  0.789858       0.816435     13.0  39.714168   
    1  26.0     260.0    1999.841182  0.164231       0.212773     13.0  17.472183   
    2   1.0      10.0   78311.447221  0.789858       0.816435     13.0  52.486000   
    3  26.0     260.0    5487.221927  0.265201       0.293801     13.0  18.667831   
    4   1.0      10.0  232299.337923  0.789858       0.816435     13.0  82.268328   
    5  26.0     260.0    1428.355411  0.125830       0.849527     13.0  23.221746   
    6   1.0      10.0   52560.966403  0.789858       0.816435     13.0  21.125350   
    7  26.0     260.0     368.076713  0.379755       0.896231      3.0  19.487683   
    8   1.0      10.0  147318.705997  0.821884       0.527160      2.0  42.190744   
    9  26.0     260.0    1869.989020  0.326712       0.924639     16.0  19.501904   
    
                 df island_#  
    0  1.912363e+06        0  
    1  8.641547e+03        0  
    2  1.148887e+06        1  
    3  4.478749e+04        1  
    4  1.952969e+06        2  
    5  3.955732e+04        2  
    6  1.345214e+06        3  
    7  4.682571e+04        3  
    8  1.114900e+06        4  
    9  5.839716e+04        4   
    

    I hope this will save someone's time while waiting for an update in the docs...