Main code below:
import hyperopt
from hyperopt import STATUS_OK
from hyperopt import tpe, hp, fmin, Trials
x_train, x_test, y_train, y_test = train_test_split(all_mags, redshifts, test_size=0.40, random_state=1)
def objective_function(params):
clf = XGBRegressor(**params)
score = cross_val_score(clf, x_train, y_train, cv=5,scoring='neg_mean_squared_error').mean()
return {'loss': score, 'status': STATUS_OK}
space= {
'learning_rate': hp.loguniform('learning_rate', np.log(0.01), np.log(1)),
'max_depth': hp.quniform('max_depth', 5, 15, 1),
'n_estimators': hp.quniform('n_estimators', 150, 250, 1),
'min_child_weight' : hp.quniform('min_child_weight', 0, 10, 1),
'gamma': hp.uniform ('gamma', 1,9),
'reg_lambda': hp.uniform('reg_lambda', 0.0, 1.0),
}
trials = Trials()
best_param = fmin(objective_function, space, algo=tpe.suggest, max_evals=500, trials=trials)
for each trial, I get this error:
as well as: "File "/Users/xxx/opt/anaconda3/lib/python3.9/site-packages/xgboost/training.py", line 178, in train for i in range(start_iteration, num_boost_round): TypeError: 'float' object cannot be interpreted as an integer"
any ideas why this is?
also for reference, I am trying to follow the tutorial here: https://www.kaggle.com/code/prashant111/bayesian-optimization-using-hyperopt. I basically just copy pasted the given code so not sure why it's not working? one thing is I got rid of the negative in the 'loss':score because the score returned with the neg method is already negative.
Figured out the issue. hp.quniform returns floats instead of ints (e.g. 2.0 instead of 2), which causes errors when indexing. Found a solution here: https://github.com/hyperopt/hyperopt/issues/253. You want to import scope (from hyperopt.pyll.base import scope) then use scope.int around all your hp.quniform (e.g. 'max_depth': scope.int(hp.quniform('max_depth', 5, 15, 1)).