rpredictstatistical-test

How to apply wilcox test on the predicted data in R?


We use the following code for k-fold cross validation training data when we have one data file,

set.seed(308)

rand_search <- train(
    Effort ~ ., data = d,
    method = "svmRadial",
    ##Create 20 random parameter values
    tuneLength = 20,
    metric = "RMSE",
    preProc = c("center", "scale"),
    trControl = rand_ctrl
) 
  model1 <- predict(rand_search, newdata = test1)

And another search algorithm like grid
grid_search <- train(
    Effort ~ ., data = d,
    method = "svmRadial",
    ##Create 20 random parameter values
    tuneLength = 20,
    metric = "RMSE",
    preProc = c("center", "scale"),
    trControl = rand_ctrl
) 
model2 <- predict(grid_search, newdata = test1)

My question is if we have to find the significance test (wilcox test), how can we apply it? Do we need to pass mode1 and model 2 to wilcox test like below?

wilcox.test(model1, model 2)


Solution

  • In trainControl you need not to specify the data. Within train function you have to mention the data like

    #Model training
    set.seed(308) 
    rand_search <- train(Effort ~ ., data = train1 ,
                                    method = "svmRadial",
                                    ## Create 20 random parameter values
                                    tuneLength = 20,
                                    metric = "RMSE",
                                    preProc = c("center", "scale"),
                                    trControl = rand_ctrl)
    

    and the test1 should be used for prediction purpose like

    #For calibration
    models_cal <- predict(rand_search, newdata = train1)
    #For independent validation
    models_val <- predict(rand_search, newdata = test1)