I am trying to run a stepwise regression using AIC (through step) with 1,400 variables, but my computer just freezes. It works if I include <300 variables (after 13 hrs of running).
Is there a way to eliminate some of the variables (if p-value >.7) before I run the stepwise regression?
# Polynomial Regression
REG19 <- lm(R10 ~ poly(M1, 3) + poly(M2, 3) + poly(M3, 3), WorkData)
# Is there a way to get rid of variables with
# p values >.7 at this point of the code?
# Beginning of stepwise regression
n <- length(resid(REG19))
REG20 <- step(REG19, direction="backward", k=log(n))
What you probably want is to exclude anything about the highest polynom where p <= .7 (lower degrees should be kept). Supposed you know what you're doing, you could write a function degAna() that analyzes the degrees of each polynomial and apply it to the coefficients matrix obtained by summary.
REG19 <- lm(R10 ~ poly(M1, 3) + poly(M2, 3) + poly(M3, 3) + poly(M4, 3) +
poly(M5, 3) + poly(M6, 3) + poly(M7, 3) + poly(M8, 3) +
poly(M9, 3) + poly(M10, 3), WorkData)
rr <- summary(REG19)$coefficients
The function that detects highest degree with p <= .7:
degAna <- function(d) {
out <- as.matrix(rr[grep(paste0(")", d), rownames(rr)), "Pr(>|t|)"] <= .7)
dimnames(out) <- list(c(gsub("^.*\\((.*)\\,.+", "\\1", rownames(out))), d)
return(out)
}
lapply degAna to coefficients matrix:
dM <- do.call(cbind, lapply(1:3, degAna)) # max. degree always 3 as in example
# 1 2 3
# M1 TRUE TRUE TRUE
# M2 TRUE TRUE TRUE
# M3 FALSE TRUE TRUE
# M4 TRUE TRUE TRUE
# M5 TRUE TRUE TRUE
# M6 TRUE FALSE TRUE
# M7 TRUE FALSE FALSE
# M8 TRUE TRUE TRUE
# M9 TRUE TRUE FALSE
# M10 TRUE FALSE TRUE
Now we need the last degree of the polynomials where p <= .7:
tM <- apply(dM, 1, function(x) max(which(x != 0)))
tM <- tM[tM > 0] # excludes polynomes where every p < .7
# M1 M2 M3 M4 M5 M6 M7 M8 M9 M10
# 3 3 3 3 3 3 1 3 2 3
(Note, that the apply will throw a warning if a polynomial has completely p <= .7, i.e. row is completely FALSE. Since we throw them out in the next line we can ignore the warning with apply(dM, 1, function(x) suppressWarnings(max(which(x != 0)))).)
With this information we can cobble together a new formula with reformulate,
terms.new <- paste0("poly(", names(tM), ", ", tM, ")")
FO <- reformulate(terms.new, response="R10")
# R10 ~ poly(M1, 3) + poly(M2, 3) + poly(M3, 3) + poly(M4, 3) +
# poly(M5, 3) + poly(M6, 3) + poly(M7, 1) + poly(M8, 3) + poly(M9,
# 2) + poly(M10, 3)
with which we finally can do the desired shortened regression.
REG19.2 <- lm(FO, WorkData)
n <- length(resid(REG19.2))
REG20.2 <- step(REG19.2, direction="backward", k=log(n))
# [...]
Simulated Data
set.seed(42)
M1 <- rnorm(1e3)
M2 <- rnorm(1e3)
M3 <- rnorm(1e3)
M4 <- rnorm(1e3)
M5 <- rnorm(1e3)
M6 <- rnorm(1e3)
M7 <- rnorm(1e3)
M8 <- rnorm(1e3)
M9 <- rnorm(1e3)
M10 <- rnorm(1e3)
R10 <- 6 + 5*M1^3 + 4.5*M2^3 + 4*M3^2 + 3.5*M4 + 3*M5 + 2.5*M6 + 2*M7 +
.5*rnorm(1e3, 1, sd=20)
WorkData <- data.frame(M1, M2, M3, M4, M5, M6, M7, M8, M9, M10, R10)