Markov Chains study in linear algebra.

1. Linear Algebra in Markov
Chains
Presented By : Luckshay Batra
luckybatra17@gmail.com

2. Introduction
Transition Matrix : A matrix in which all entries are
nonnegative and all columns sum to 1.
Thus, an n x n Transition matrix gives the probability of
movement from any one of n states to any of the other n
states.
Probability Vector : A vector whose coordinates are
nonnegative and sum to 1. Thus, an n-dimensional
probability vector gives the chances of occurrence of
each one of n events.

3. Markov Chain
 If p0 is a probability vector and A is a transition matrix, then the sequence,
p0, p1, p2, p3, ... , where pk = (A^k)*p0 for k = 1, 2, 3, ..., is called a Markov
chain.
 If A*p = p for some probability vector, then p is called a steady-state
vector.
 A Stochastic model that describes the probabilities of transition among
the states of a system.
 Change of state depends probabilistically only on the current state of the
system.
 Independent of the past given that the present state is known.
 The behavior depends on the structure of the transition matrix P.

4. Problem of Television Viewers
 Suppose there are two regional news shows in the local
television viewing area, and we have conducted a survey of
viewers to determine which channel the viewers have been
watching. The first survey revealed that 40% of the viewers
watched station X and 60% watched station Y. Subsequent
surveys revealed that each week 15% of the X viewers
switched to station Y and 5% of the Y viewers switched to
station X.
 We will use Transition matrices and Markov chains to make a
prediction about the future television market from this
information.

5.  We assume that the 15%-5% switching trends will continue
indefinitely.
 Let pk be a two-dimensional column vector whose entries give
the proportion of people who watch station X and the
proportion who watch station Y, in that order, during week k.
Thus, p0 = [0.4,0.6]t.
From the assumptions above, we see that the proportion of
people watching station X in week k = 1 will be 85% of the X
viewers in week k = 0 (which is 85% of 40%, or 34%) plus 5%
of the Y viewers in week k = 0 (which is 5% of 60%, or 3%) for
a total of 37%:
0.85(0.4) + 0.05(0.6) = 0.37.

6. Similarly,
0.15(0.4) + 0.95(0.6) = 0.63,
is the proportion of people watching station Y in week k = 1. So,
we have computed p1 = [0.37,0.63]t.
Similarly, we could compute the sequence of vectors p2, p3, ... ,
but this process would be quite tedious if we continued in the
manner above.
Andrei Markov (1856-1922) described a much more efficient
way of handling such a problem.

7. First, we think of the movement of viewers as being described by the
following array, which gives the weekly proportion of viewers who change
from one station to another:
From:
The matrix A of entries in the table is called the transition matrix for this
problem.
Looking carefully at the definition of matrix multiplication, we see that
p1=A*p0. Indeed, we see that, for k = 0, 1, 2, ..., and so on, pk+1=A*pk.
Each pk is called a probability vector, and the sequence, p0, p1, p2, p3, ..., is
called a Markov chain.

8. Because matrix multiplication is associative, we find
p2 = A*p1 = A(A*p0) = (A^2)*p0,
p3 = A*p2 = A((A^2)*p0) = (A^3)*p0,
and, in general,
pk = (A^k)*p0, for k = 1, 2, ...
Compute A^5. Then compute p5 directly from A^5 without
computing p1, ... , p4.

9. Suppose p is a probability vector with the property that A*p = p.
If this p describes the current viewers .
We can solve the equation A*p = p by rewriting it as A*p - p = 0 and factoring
out p.
We get the matrix equation (A - I)p = 0, where I is the 2x2 identity matrix.
Using our given transition matrix, solve this matrix equation for p.
From the solutions pick the one which is a probability vector (i.e., its entries
must sum to 1).
This vector is called the steady-state vector.
The fact that p is the steady-state vector does not imply that the Markov chain
will actually reach that state from p0.

10. References
• Statistics & Probability by Sheldon M Ross
• Linear Algebra by David C Lay