ralgorithmglmscoring

How do I construct a diagonal vector that has entries exceed the maximum number of entries allowed in a matrix in R?

I wish to construct X^TWX of a large dataset with 67856 elements.

I am trying to construct the diagonal vector, W of dimension 67856*67856 but I am running into this error "vector memory exhausted (limit reached?)"

Are there any alternative methods to construct the diagonal vector W which is needed for my Fischer Scoring algorithm to work

initial_model1=(lm(ClaimNb~VehValue+VehAge+DrivAge,data=final_data))
r=coef(summary(mleNB))[1,1]
beta0=initial_model1$coefficients[1]
beta1=initial_model1$coefficients[2]
beta2=initial_model1$coefficients[3]
beta3=initial_model1$coefficients[4]
y=c(final_data$ClaimNb)
xi1=c(final_data$VehValue)
xi2=c(final_data$VehAge)
xi3=c(final_data$DrivAge)

##Column vector of regressors
regressors=matrix(c(beta0,beta1,beta2,beta3),ncol=1)
n=nrow(final_data)
beta0=regressors[1]
beta1=regressors[2]
beta2=regressors[3]
beta3=regressors[4]
intercept=rep(1,nrow(final_data))
X=matrix(c(intercept,xi1,xi2,xi3),ncol=4)
XT=t(X)
W=matrix(rep(0),n,n)
Error: vector memory exhausted (limit reached?)
Solution

  • Instead of filling the diagonal matrix, just use its diagonal as below:

    # diagonal matrix
D <- rbind(
  c(2, 0, 0),
  c(0, 3, 0),
  c(0, 0, 4)
)

# a 3x2-matrix
set.seed(666)
X <- matrix(rpois(6, 50), ncol = 2)

# brutal matricial product
t(X) %*% D %*% X

# other way
v <- diag(D) # v = c(2, 3, 4)
crossprod(sqrt(v) * X) # = t(X) %*% D %*% X