I have the following states
states = [(0,2,3,0), (2,2,3,0), (2,2,2,0), (2,2,1,0)]
In addition, I have the following transition matrix
import pandas as pd
transition_matrix = pd.DataFrame([[1, 0, 0, 0],
[0.5, 0.3, 0.2, 0],
[0.5, 0.3, 0, 0.2],
[0.5, 0.5, 0, 0]], columns=states, index=states)
So, if you are in state (2,2,1,0) there is a 50% that you go to state (0,2,3,0) and a 50% probability that you go (2,2,3,0).
If you are in state (0,2,3,0), the absorbing state, you win.
We can write the following equations
p_win_(0,2,3,0) = 1 p_win_(2,2,3,0) = 0.50 * p_win_(0,2,3,0) + 0.3 * p_win(2,2,3,0) + 0.2 * p_win(2,2,2,0) p_win_(2,2,2,0) = 0.50 * p_win_(0,2,3,0) + 0.3 * p_win(2,2,3,0) + 0.2 * p_win(2,2,1,0) p_win_(2,2,1,0) = 0.50 * p_win_(0,2,3,0) + 0.5 * p_win(2,2,3,0)
I would like to solve the above formulas. I looked at the documentation of the np.linalg.solve-function. The example doesn't use defined variables and, in addition, I have terms on both side of the equal sign.
Please show me how I can solve the above.
First, your first equation is wrong (it should be p_win_(0,2,3,0) = 1* p_win_(0,2,3,0))
You are essentially trying to get the largest eigenvector (corresponding to eig=1) for the Transition matrix. p_win_ is determined by:
v = Pv (or P-I)v, sum(v) = 1, where I is identity matrix np.eye(4)
which we can write in extended form as:
I = np.eye(4)
P = np.array([[1, 0, 0, 0],
[0.5, 0.3, 0.2, 0],
[0.5, 0.3, 0, 0.2],
[0.5, 0.5, 0, 0]]) # if you already have it in DataFrame,
# you can alternatively do:
# P = transition_matrix.to_numpy()
extend_m = np.concatenate((P-I, np.ones((1,4), axis=0))
# Equation to solve is extend_m*v = np.array([0,1])
So solution is given by
v = np.linalg.lstsq(extend_m, np.array([0,1])
I use lstsq because we have an overdetermined system (5 equations, 4 unknowns). If you want to use np.linalg.solve you need to reduce it to 4 equations, which I leave up to you (In this particular case, there is one obviously redundant equation, which you can just remove).