We use the following code for k-fold cross validation training data when we have one data file,
set.seed(308)
rand_search <- train(
Effort ~ ., data = d,
method = "svmRadial",
##Create 20 random parameter values
tuneLength = 20,
metric = "RMSE",
preProc = c("center", "scale"),
trControl = rand_ctrl
)
model1 <- predict(rand_search, newdata = test1)
And another search algorithm like grid
grid_search <- train(
Effort ~ ., data = d,
method = "svmRadial",
##Create 20 random parameter values
tuneLength = 20,
metric = "RMSE",
preProc = c("center", "scale"),
trControl = rand_ctrl
)
model2 <- predict(grid_search, newdata = test1)
My question is if we have to find the significance test (wilcox test), how can we apply it? Do we need to pass mode1 and model 2 to wilcox test like below?
wilcox.test(model1, model 2)
In trainControl
you need not to specify the data. Within train
function you have to mention the data like
#Model training
set.seed(308)
rand_search <- train(Effort ~ ., data = train1 ,
method = "svmRadial",
## Create 20 random parameter values
tuneLength = 20,
metric = "RMSE",
preProc = c("center", "scale"),
trControl = rand_ctrl)
and the test1
should be used for prediction purpose like
#For calibration
models_cal <- predict(rand_search, newdata = train1)
#For independent validation
models_val <- predict(rand_search, newdata = test1)