rmlr3

Change the resampling of an auto_tuner in mlr3


I am using mlr3 and I wanted to ask if it is possible to change the resampling method of an exiting auto_tuner().

Example:

library(mlr3verse)
# Some existing auto_tuner

learner = lrn("classif.svm",
  cost  = to_tune(1e-1, 1e5),
  gamma = to_tune(1e-1, 1),
  kernel = "radial",
  type = "C-classification"
)

at = auto_tuner(
  tuner = tnr("grid_search", resolution = 5, batch_size = 5),
  learner = learner,
  resampling = rsmp("cv", folds = 3), # The resampling I would like to change
  measure = msr("classif.ce")
)

# New resampling I would like to assign to the existing auto_tuner

new_resampling = rsmp("cv", folds = 10)

Background:

I select a model based on a nested cross validation and afterwards want to train the best model for prediction. As I use a simpler resampling inside my nested cross validatio I would like to change the resampling used by the auto_tuner to avoid creating a new one.


Solution

  • I select a model based on a nested cross validation

    Maybe you should think again about your approach:

    Nested resampling is a statistical procedure to estimate the predictive performance of the model trained on the full dataset, it is not a procedure to select optimal hyperparameters. Nested resampling produces many hyperparameter configurations which should not be used to construct a final model

    mlr3book

    Regarding your question: No, it is not possible to change the resampling later. Just construct a new one if you want to change the resampling.