rmatrixtransitionchainmarkov

Is there an R package to calculate 1st order transition matrix from a frequency table?


I have a frequency table aggregated from 800 millions of records and am wondering if I can use a package to calculate 1st order transition matrix from the frequency table, which is not symmetric because some state just never happened again. A sample of the frequency table is:

library(data.table)
model.data <- data.table(state1 = c(3, 1, 2, 3), state2 = c(1, 2, 1, 2), Freq = c(1,2,3,4))

model.data looks like this:

state1 state2 n
3 1 1
1 2 2
2 1 3
3 2 4

Using the package pollster, I can compute the proportion table:

library(pollster)
crosstab(model.data, state1, state2, Freq)
state1 1 2 n
1 0 100 2
2 100 0 3
3 20 80 5

However, the symmetric transition matrix I am looking for is:

state1 1 2 3 n
1 0 100 0 2
2 100 0 0 3
3 20 80 0 5

That is, I still want to include the state 3 even though no one transitioned to it, and the code should be able to automatically find out 3 needs to be appended with a column of 0s.

I am not sure if the markovchain package with the markovchainFit function is going to handle my 800 million rows of data that I need to transform into a list of millions of sequences, due to memory constraints and slow computing speed.

Does anyone know?


Solution

  • An option with igraph

    model.data %>%
      setorder(state1) %>%
      graph_from_data_frame() %>%
      as_adjacency_matrix(attr = "Freq", sparse = FALSE) %>%
      proportions(1)  # 1 sets rows as the margin, similar to `prop.table`
    

    gives

        1   2 3
    1 0.0 1.0 0
    2 1.0 0.0 0
    3 0.2 0.8 0
    

    Or with base R

    > proportions(xtabs(Freq ~ ., model.data), 1)
          state2
    state1   1   2
         1 0.0 1.0
         2 1.0 0.0
         3 0.2 0.8