I'm new to R and I've ran into this problem: I want to compare two prediction techniques (Support Vector Machines and Neural Networks) applying them to some data and I would like to compare their performance. To do this, I use ROC curves. The code is supposed to compute the area under the ROC curve but it is not working. Neural networks code works fine, but when SVM part executes there was this error:
> aucs <- auc((dtest$recid=="SI")*1, lr.pred)
Error in roc.default(response, predictor, auc = TRUE, ...) : Predictor must be numeric or ordered.
> obj.roc <- roc((dtest$recid=="SI")*1, lr.pred )
Error in roc.default((dtest$recid == "SI") * 1, lr.pred) : Predictor must be numeric or ordered.
This is the code I have.
library(stats)
library(pROC)
library(nnet)
library(e1071)
library(rpart)
data <- read.table("data.csv", header=T)
set.seed(1234)
ind <- sample(2, nrow(data), replace=TRUE, prob=c(0.8, 0.2))
dtrain <- data[ind==1,]
dtest <- data[ind==2,]
# Variables for storing comparison results #
bestAuc = 0
bestIdx = 0
# Support Vector Machines
lr.fit <- svm(recid~., data=dtrain, cost=1000, gamma=1, probability=TRUE)
lr.pred <- predict(lr.fit, dtest, type="response")
aucs <- auc((dtest$recid=="SI")*1, lr.pred)
obj.roc <- roc((dtest$recid=="SI")*1, lr.pred)
print("SVN (default)")
bestAuc = aucs # Initialize
# Neural networks
lr.fit <- nnet(recid~., data=dtrain, size=4, maxit=500, decay=1, trace=FALSE)
lr.pred <- predict(lr.fit, dtest, type="raw")
aucs <- auc((dtest$recid=="SI")*1, lr.pred)
obj.roc <- roc((dtest$recid=="SI")*1, lr.pred )
if(aucs > bestAuc) {
bestAuc <- aucs
bestIdx <- 1
print("Neural networks")
}
I've been looking for information but it seems that there is little about the methods I'm using. I saw a package called ROCR which I think could be useful but I also get errors with the performance function. I'm a little bit lost with all this libraries so I tried to stick with my initial solution with no improvements. What should I do?
EDIT:
The solution was based on the idea of Calimo. The return value of predict does not give the data in the format I wanted so I needed to use this:
lr.pred <- attr(lr.pred,"probabilities")[,c("SI")]
The sentence gets the column that is going to be analyzed in the ROC curve.
As the error message say, you need a numeric vector or ordered factor in lr.pred. The problem here is that predict (for the svm) returns the predicted class, making the ROC exercise pretty much useless.
What you need is to get an internal score, like the class probabilities:
lr.pred <- predict(lr.fit, dtest, probability = TRUE)
(You will have to choose which probability to get, for the first or second class. Also note that type = "response" is ignored.)