I have a frequency table aggregated from 800 millions of records and am wondering if I can use a package to calculate 1st order transition matrix from the frequency table, which is not symmetric because some state just never happened again. A sample of the frequency table is:
library(data.table)
model.data <- data.table(state1 = c(3, 1, 2, 3), state2 = c(1, 2, 1, 2), Freq = c(1,2,3,4))
model.data looks like this:
state1 | state2 | n |
---|---|---|
3 | 1 | 1 |
1 | 2 | 2 |
2 | 1 | 3 |
3 | 2 | 4 |
Using the package pollster, I can compute the proportion table:
library(pollster)
crosstab(model.data, state1, state2, Freq)
state1 | 1 | 2 | n |
---|---|---|---|
1 | 0 | 100 | 2 |
2 | 100 | 0 | 3 |
3 | 20 | 80 | 5 |
However, the symmetric transition matrix I am looking for is:
state1 | 1 | 2 | 3 | n |
---|---|---|---|---|
1 | 0 | 100 | 0 | 2 |
2 | 100 | 0 | 0 | 3 |
3 | 20 | 80 | 0 | 5 |
That is, I still want to include the state 3 even though no one transitioned to it, and the code should be able to automatically find out 3 needs to be appended with a column of 0s.
I am not sure if the markovchain package with the markovchainFit function is going to handle my 800 million rows of data that I need to transform into a list of millions of sequences, due to memory constraints and slow computing speed.
Does anyone know?
An option with igraph
model.data %>%
setorder(state1) %>%
graph_from_data_frame() %>%
as_adjacency_matrix(attr = "Freq", sparse = FALSE) %>%
proportions(1) # 1 sets rows as the margin, similar to `prop.table`
gives
1 2 3
1 0.0 1.0 0
2 1.0 0.0 0
3 0.2 0.8 0
Or with base R
> proportions(xtabs(Freq ~ ., model.data), 1)
state2
state1 1 2
1 0.0 1.0
2 1.0 0.0
3 0.2 0.8