pythonmachine-learninghyperparametersoptuna

Is there a way for Optuna `suggest_categorical`to return multiple choices from list?


I am using Optuna for hyperparametrization of my model. And I have a field where I want to test multiple combinations from a list. For example: I have ["lisa","adam","test"] and I want suggest_categorical to return not just one, but a random combination: maybe ["lisa", "adam"], maybe ["adam"], maybe ["lisa", "adam", "test"]. Is there a way to get this with built in Optuna function?


Solution

  • You could use itertools.combinations to generate all possible combinations of list items and then pass them to optuna's suggest_categorical as choices:

    import optuna
    import itertools
    import random
    import warnings
    warnings.filterwarnings('ignore')
    
    # generate the combinations
    iterable = ['lisa', 'adam', 'test']
    combinations = []
    for r in range(1, len(iterable) + 1):
        combinations.extend([list(x) for x in itertools.combinations(iterable=iterable, r=r)])
    print(combinations)
    # [['lisa'], ['adam'], ['test'], ['lisa', 'adam'], ['lisa', 'test'], ['adam', 'test'], ['lisa', 'adam', 'test']]
    
    # sample the combinations
    def objective(trial):
        combination = trial.suggest_categorical(name='combination', choices=combinations)
        return round(random.random(), 2)
    
    study = optuna.create_study()
    study.optimize(objective, n_trials=3)
    # [I 2022-08-18 08:03:51,658] A new study created in memory with name: no-name-3874ce95-2394-4526-bb19-0d9822d7e45c
    # [I 2022-08-18 08:03:51,659] Trial 0 finished with value: 0.94 and parameters: {'combination': ['adam']}. Best is trial 0 with value: 0.94.
    # [I 2022-08-18 08:03:51,660] Trial 1 finished with value: 0.87 and parameters: {'combination': ['lisa', 'test']}. Best is trial 1 with value: 0.87.
    # [I 2022-08-18 08:03:51,660] Trial 2 finished with value: 0.29 and parameters: {'combination': ['lisa', 'adam']}. Best is trial 2 with value: 0.29.
    

    Using lists as choices in optuna's suggest_categorical throws a warning message, but apparently this is mostly inconsequential (see this issue in optuna's GitHub repository).