rr-bigmemory

Parallel big matrix multiplication


I need to multiply two big matrices A and B as follow:

library(bigmemory)
library(bigalgebra)
library(biganalytics)

A <- big.matrix( replicate(100,  rnorm(10^5)) )
B <- big.matrix( replicate(10^5, rnorm(100))  )
AB <- A %*% B

How could I compute this multiplication in parallel?

The only tutorial I've come across so far is this one:

> library("doRedis")
> registerDoRedis(queue="example")
> L = foreach(j=1:2,.packages="VAM",.combine=c) %dopar%
+ {
+ key = paste("X",j,sep="")
+ ridx = ((j-1)*5 + 1):min((j*5),nrow(A))
+ X = A[ridx,] %*% B[,]
+ Y = as.big.matrix(X,backingfile=key)
+ vnew(Y, key)
+ key
+ }
> X = vam(matrix(L,nrow=2))
> sum(X[,] - A[,] %*% B[,])
[1] 0

But I'm not sure how to put it into practice. There may also be a simpler/more efficient way to achieve the same result?


Solution

  • Installing Microsoft R Open, I go from 3 sec to 0.1 sec!

    library(bigmemory)
    library(bigalgebra)
    
    N <- 200
    M <- 1e5
    
    A <- big.matrix(N, M, init = rnorm(N * M))
    B <- big.matrix(M, N, init = rnorm(N * M))
    system.time(AB <- A %*% B)
    

    Change the version of R you use in RStudio settings (need to quit RStudio afterwards)