Working on parameter tuning an xgboost model, and ran into an interesting error in my implementation of mlr, which I believe caused by my resample instance due to the documentation here. the problem is I can't quite track down how to fix it. I've tried manually setting the size param for the function, but that is also rejected.
basic code:
samplecount = sample.split(test_train,SplitRatio = 0.8)
stest <- subset(test_train,samplecount ==TRUE)
strain <- subset(test_train,samplecount ==FALSE)
new_tr <- model.matrix(~.+0,data = subset(strain,select=-c(Value)))
new_ts <- model.matrix(~.+0,data = subset(stest,select=-c(Value)))
labels <- strain$Value
labels <- as.numeric(labels)-1
ts_label <- stest$Value
ts_label <- as.numeric(ts_label)-1
dtrain <- xgb.DMatrix(data = new_tr,label = labels)
dtest <- xgb.DMatrix(data = new_ts,label=ts_label)
params <- list(booster = "gbtree", objective = "reg:linear",
eta=0.3, gamma=0, max_depth=6, min_child_weight=1, subsample=1, colsample_bytree=1)
xgbcv <- xgb.cv( params = params, data = dtrain,
nrounds = 100, showsd = T,nfold=5,
stratified = T, print.every.n = 1, early.stop.round = 20, maximize = F)
xgb1 <- xgb.train (params = params, data = dtrain, nrounds = 79, watchlist = list(val=dtest,train=dtrain), print.every.n = 10, early.stop.round = 10, maximize = F , eval_metric = "error")
xgbpred <- predict (xgb1,dtest)
mat <- xgb.importance (feature_names = colnames(new_tr),model = xgb1)
xgb.plot.importance (importance_matrix = mat[1:20])
#convert characters to factors
fact_col <- colnames(strain)[sapply(strain,is.character)]
for(i in fact_col) set(strain,j=i,value = factor(strain[[i]]))
for (i in fact_col) set(stest,j=i,value = factor(stest[[i]]))
## this seems like an odd add, but for the steps
strain$Value <- as.factor(strain$Value)
stest$Value <- as.factor(stest$Value)
#create tasks
traintask <- makeClassifTask (data = strain,target = "Value",fixup.data = "no")
testtask <- makeClassifTask (data = stest,target = "Value",fixup.data = "no")
#do one hot encoding`<br/>
traintask <- createDummyFeatures (obj = traintask)
testtask <- createDummyFeatures (obj = testtask)
lrn <- makeLearner("regr.xgboost",predict.type = "response")
lrn$par.vals <- list( objective="reg:linear", eval_metric="error", nrounds=100L, eta=0.1)
#set parameter space
params <- makeParamSet( makeDiscreteParam("booster",values = c("gbtree","gblinear")),
makeIntegerParam("max_depth",lower = 3L,upper = 10L),
makeNumericParam("min_child_weight",lower = 1L,upper = 10L),
makeNumericParam("subsample",lower = 0.5,upper = 1),
makeNumericParam("colsample_bytree",lower = 0.5,upper = 1))
#set resampling strategy
rdesc <- makeResampleDesc("CV",stratify = T,iters=5L)
ctrl <- makeTuneControlRandom(maxit = 10L)
library(parallel)
library(parallelMap)
parallelStartSocket(cpus = detectCores())
#parameter tuning
mytune <- tuneParams(learner = lrn, task = traintask, resampling = rdesc, , measures = acc, par.set = params, control = ctrl, show.info = T)
Error in instantiateResampleInstance.CVDesc(desc, length(ci), task) :
Cannot use more folds (5) than size (1)!
From there I tried:
rdesc <- makeResampleDesc("CV",stratify = T,size=5)
Error in makeResampleDescCV(size = 5) : unused argument (size = 5)
I'm at a bit of a loss here, any ideas?
Size is not a parameter in makeResampleDesc. I think (not completely sure), that your problem is, that you do not have enough observations for certain classes and then you cannot do stratification.
Try using: rdesc <- makeResampleDesc("CV",stratify = F,iters=5)