xgboosthyperparametershyperoptxgbregressor

using hyperopt to optimize XGBRegressor giving type error and not returning best hyperparameters


Main code below:

import hyperopt
from hyperopt import STATUS_OK
from hyperopt import tpe, hp, fmin, Trials

x_train, x_test, y_train, y_test = train_test_split(all_mags, redshifts, test_size=0.40, random_state=1)

def objective_function(params):
    clf = XGBRegressor(**params)
    score = cross_val_score(clf, x_train, y_train, cv=5,scoring='neg_mean_squared_error').mean()
    return {'loss': score, 'status': STATUS_OK}   

space= {
    'learning_rate': hp.loguniform('learning_rate', np.log(0.01), np.log(1)),
    'max_depth': hp.quniform('max_depth', 5, 15, 1),
    'n_estimators': hp.quniform('n_estimators', 150, 250, 1),
    'min_child_weight' : hp.quniform('min_child_weight', 0, 10, 1),
    'gamma': hp.uniform ('gamma', 1,9),
    'reg_lambda': hp.uniform('reg_lambda', 0.0, 1.0),
}

trials = Trials()
best_param = fmin(objective_function, space, algo=tpe.suggest, max_evals=500, trials=trials)

for each trial, I get this error: trial error

as well as: "File "/Users/xxx/opt/anaconda3/lib/python3.9/site-packages/xgboost/training.py", line 178, in train for i in range(start_iteration, num_boost_round): TypeError: 'float' object cannot be interpreted as an integer"

any ideas why this is?

also for reference, I am trying to follow the tutorial here: https://www.kaggle.com/code/prashant111/bayesian-optimization-using-hyperopt. I basically just copy pasted the given code so not sure why it's not working? one thing is I got rid of the negative in the 'loss':score because the score returned with the neg method is already negative.


Solution

  • Figured out the issue. hp.quniform returns floats instead of ints (e.g. 2.0 instead of 2), which causes errors when indexing. Found a solution here: https://github.com/hyperopt/hyperopt/issues/253. You want to import scope (from hyperopt.pyll.base import scope) then use scope.int around all your hp.quniform (e.g. 'max_depth': scope.int(hp.quniform('max_depth', 5, 15, 1)).