rmcmcpscl

ideal() in R package pscl not producing repeatable results


I'm working with the pscl package in R and trying to get it to produce testable/reproducible results. I've taken a look at the underlying C code and it appears as though GetRNGstate() and PutRNGstate() are being called in the right places but it seems impossible to repeat output from the MCMC model.

I've packaged up the functions in simulationResult from the SoDA package so I can verify the start state of each simulation R on the R side.

library(pscl)
library(SoDA)
run1 <- simulationResult(
  ideal(s109, 
    normalize=TRUE,
    maxiter = 500,
    thin = 10,
    burnin = 0),
  seed = 42)

run2 <- simulationResult(
  ideal(s109, 
    normalize=TRUE,
    maxiter = 500,
    thin = 10,
    burnin = 0),
  seed = 42)

We can verify that the start states are the same at least on the R side:

all.equal(run1@firstState, run2@firstState)

But the output is different:

all.equal(run1@result$xbar, run2@result$xbar)

I can increase the number of iterations but that shouldn't really matter if the RNG state is getting propogated. Am I missing something really simple? Thanks.

Edit: I should also note that all.equal(run1@lastState, run2@lastState) (the end state of each run) should be the same but they end up different. My guess is that some source of contingency outside of the R RNG functions called by C is impacting the number of times those RNG functions are called. Curious.

Edit2

I should also add I'm on R 3.0.1 with pscl 1.04.4 on OS X 10.8.4.


Solution

  • As suspected by OP and @SchaunW, the problem lies in the C code. "A bit" of digging revealed a quite subtle problem (see the source code, not the newest version though):

    All the sampling in ideal.c appears in the part of commence iterations, i.e. where functions updatex, updatey and others are used. However, the problem is with one of the arguments of these functions - the matrix ok (ironic, right?). It is used by updatex and updateb and only the positions where ok == 1 are important (in crosscheck, crosscheckx).

    Before that, some values of ok are assigned to be 1 in check(y,ok,n,m).

    However, at the very beginning the initial values of ok are denoted by

    ok = imatrix(n,m);
    

    which allocates a integer matrix (see util.c for imatrix). The problem is that then ok contains various numbers, i.e. not only zeros and sometimes ones. It seems that they are not related to RNG state of R, which explains the behaviour noted by @SchaunW: all.equal(run1@result$xbar, run2@result$xbar) returns TRUE if !any(ok == 1) and vice versa. Also, different number of ones explains different lastState.

    I am not an expert in C and I am not sure whether there is a logic error in the code or if the imatrix function should be corrected, but a straightforward fix could be to fill ok with zeros right after initialisation:

    ok = imatrix(n,m);
    for(a=0; a<n; a++) {
        for(aa=0; aa<m; aa++) {
          ok[a][aa] = 0;
        }
    }
    

    Finally, there is also a fix that does not include modifying the C code (it might not be right for your applications though). Functions crossxyi, crossxyj are used instead of crosscheck, crosscheckx (the bad ones) when impute = TRUE for ideal.