rr-bigmemoryrparallel

addressing a big.matrix from cluster workers

I am trying to access a big.matrix (not file backed) from a parLapply function on a windows machine. However, R crashes when calling the big.matrix - "R for Windows front-end stopped working".

Do I need to attach the big.matrix first? How do I do this? Any help is highly appreciated.

require(parallel)
require(bigmemory)

data <- matrix(rnorm(10^8),ncol=1000)
data.big <- as.big.matrix(data)

cl <- makeCluster(2)

parLapply(cl,1:2,function(x,data.big){
  require(bigmemory)
  data.big[x,1] # this line causes R to crash
},data.big)

stopCluster(cl)
Solution

  • You want to use attach.big.matrix to access the shared memory. This is done by using the information from describe. The following should work.

    datadesc <- describe(data.big)

parLapply(cl,1:2,function(x,datadesc){
  require(bigmemory)
  data.big <- attach.big.matrix(datadesc)
  data.big[x,1] # this line causes R to crash
},datadesc)