Upcoming SlideShare
×

# Lesson 11: Markov Chains

11,674 views

Published on

Markov chains are a very common model for systems that change probablistically over time. We show a few fun examples, define the objects, state the main theorems, and show how to find the steady-state vector.

Published in: Technology, Education
• Full Name
Comment goes here.

Are you sure you want to Yes No
• \$10 Risk Free Spins GO HERE https://t.co/IzgLwbyMqx

Are you sure you want to  Yes  No
• Thank you for sharing

Are you sure you want to  Yes  No
• a WONDERFUL SLIDE SHOW FOR MARKOV MODEL

Are you sure you want to  Yes  No

### Lesson 11: Markov Chains

1. 1. Lesson 11 Markov Chains Math 20 October 15, 2007 Announcements Review Session (ML), 10/16, 7:30–9:30 Hall E Problem Set 4 is on the course web site. Due October 17 Midterm I 10/18, Hall A 7–8:30pm OH: Mondays 1–2, Tuesdays 3–4, Wednesdays 1–3 (SC 323) Old exams and solutions on website
2. 2. The Markov Dance Divide the class into three groups A, B, and C .
3. 3. The Markov Dance Divide the class into three groups A, B, and C . Upon my signal: 1/3of group A goes to group B, and 1/3 of group A goes to group C .
4. 4. The Markov Dance Divide the class into three groups A, B, and C . Upon my signal: 1/3of group A goes to group B, and 1/3 of group A goes to group C . 1/4of group B goes to group A, and 1/4 of group A goes to group C .
5. 5. The Markov Dance Divide the class into three groups A, B, and C . Upon my signal: 1/3of group A goes to group B, and 1/3 of group A goes to group C . 1/4of group B goes to group A, and 1/4 of group A goes to group C . 1/2 of group C goes to group B.
6. 6. Another Example Suppose on any given class day you wake up and decide whether to come to class. If you went to class the time before, you’re 70% likely to go today, and if you skipped the last class, you’re 80% likely to go today.
7. 7. Another Example Suppose on any given class day you wake up and decide whether to come to class. If you went to class the time before, you’re 70% likely to go today, and if you skipped the last class, you’re 80% likely to go today. Some questions you might ask are: If I go to class on Monday, how likely am I to go to class on Friday?
8. 8. Another Example Suppose on any given class day you wake up and decide whether to come to class. If you went to class the time before, you’re 70% likely to go today, and if you skipped the last class, you’re 80% likely to go today. Some questions you might ask are: If I go to class on Monday, how likely am I to go to class on Friday? Assuming the class is inﬁnitely long (the horror!), approximately what portion of class will I attend?
9. 9. Many times we are interested in the transition of something between certain “states” over discrete time steps. Examples are movement of people between regions states of the weather movement between positions on a Monopoly board your score in blackjack
10. 10. Many times we are interested in the transition of something between certain “states” over discrete time steps. Examples are movement of people between regions states of the weather movement between positions on a Monopoly board your score in blackjack Deﬁnition A Markov chain or Markov process is a process in which the probability of the system being in a particular state at a given observation period depends only on its state at the immediately preceding observation period.
11. 11. Common questions about a Markov chain are: What is the probability of transitions from state to state over multiple observations? Are there any “equilibria” in the process? Is there a long-term stability to the process?
12. 12. Deﬁnition Suppose the system has n possible states. For each i and j, let tij be the probability of switching from state j to state i. The matrix T whose ijth entry is tij is called the transition matrix.
13. 13. Deﬁnition Suppose the system has n possible states. For each i and j, let tij be the probability of switching from state j to state i. The matrix T whose ijth entry is tij is called the transition matrix. Example The transition matrix for the skipping class example is 0.7 0.8 T= 0.3 0.2
14. 14. The big idea about the transition matrix reﬂects an important fact about probabilities: All entries are nonnegative. The columns add up to one. Such a matrix is called a stochastic matrix.
15. 15. Deﬁnition The state vector of a Markov process with n-states at time step k is the vector  (k)  p 1  p (k)  x(k) =  2  . . . (k) pn (k) where pj is the probability that the system is in state j at time step k.
16. 16. Deﬁnition The state vector of a Markov process with n-states at time step k is the vector  (k)  p 1  p (k)  x(k) =  2  . . . (k) pn (k) where pj is the probability that the system is in state j at time step k. Example Suppose we start out with 20 students in group A and 10 students in groups B and C . Then the initial state vector is x(0) =
17. 17. Deﬁnition The state vector of a Markov process with n-states at time step k is the vector  (k)  p 1  p (k)  x(k) =  2  . . . (k) pn (k) where pj is the probability that the system is in state j at time step k. Example Suppose we start out with 20 students in group A and 10 students in groups B and C . Then the initial state vector is   0.5 x(0) = 0.25 . 0.25
18. 18. Example Suppose after three weeks of class I am equally likely to come to class or skip. Then my state vector would be x(10) =
19. 19. Example Suppose after three weeks of class I am equally likely to come to 0.5 class or skip. Then my state vector would be x(10) = 0.5
20. 20. Example Suppose after three weeks of class I am equally likely to come to 0.5 class or skip. Then my state vector would be x(10) = 0.5 The big idea about state vectors reﬂects an important fact about probabilities: All entries are nonnegative. The entries add up to one. Such a vector is called a probability vector.
21. 21. Lemma Let T be an n × n stochastic matrix and x an n × 1 probability vector. Then T x is a probability vector.
22. 22. Lemma Let T be an n × n stochastic matrix and x an n × 1 probability vector. Then T x is a probability vector. Proof. We need to show that the entries of T x add up to one. We have   n n n n n (T x)i = tij xj  = tij xj  i=1 i=1 j=1 j=1 i=1 n 1 · xj = 1 = j=1
23. 23. Theorem If T is the transition matrix of a Markov process, then the state vector x(k+1) at the (k + 1)th observation period can be determined from the state vector x(k) at the kth observation period, as x(k+1) = T x(k)
24. 24. Theorem If T is the transition matrix of a Markov process, then the state vector x(k+1) at the (k + 1)th observation period can be determined from the state vector x(k) at the kth observation period, as x(k+1) = T x(k) This comes from an important idea in conditional probability: P(state i at t = k + 1) n = P(move from state j to state i)P(state j at t = k) j=1 That is, for each i, n (k+1) (k) pi = tij pj j=1
25. 25. Illustration Example How does the probability of going to class on Wednesday depend on the probabilities of going to class on Monday? (k) (k) p2 p1 go Monday skip t21 t11 t12 t22 go go Wednesday skip skip (k+1) (k) (k) p1 = t11 p1 + t12 p2 (k+1) (k) (k) p2 = t21 p1 + t22 p2
26. 26. Example If I go to class on Monday, what’s the probability I’ll go to class on Friday?
27. 27. Example If I go to class on Monday, what’s the probability I’ll go to class on Friday? Solution 1 We have x(0) = . We want to know x(2) . We have 0 x(2) = T x(1) = T (T x(0) ) = T 2 = T x(0) 2 0.7 0.8 1 0.7 0.8 0.7 0.73 = = = 0.3 0.2 0 0.3 0.2 0.3 0.27
28. 28. Let’s look at successive powers of the probability matrix. Do they converge? To what?
29. 29. Let’s look at successive powers of the transition matrix in the Markov Dance.   0.333333 0.25 0. T = 0.333333 0.5 0.5 0.333333 0.25 0.5
30. 30. Let’s look at successive powers of the transition matrix in the Markov Dance.   0.333333 0.25 0. T = 0.333333 0.5 0.5 0.333333 0.25 0.5   0.194444 0.208333 0.125 T 2 = 0.444444 0.458333 0.5  0.361111 0.333333 0.375
31. 31. Let’s look at successive powers of the transition matrix in the Markov Dance.   0.333333 0.25 0. T = 0.333333 0.5 0.5 0.333333 0.25 0.5   0.194444 0.208333 0.125 T 2 = 0.444444 0.458333 0.5  0.361111 0.333333 0.375   0.175926 0.184028 0.166667 T 3 = 0.467593 0.465278 0.479167 0.356481 0.350694 0.354167
32. 32.   0.17554 0.177662 0.175347 T 4 = 0.470679 0.469329 0.472222 0.353781 0.353009 0.352431
33. 33.   0.17554 0.177662 0.175347 4 0.470679 0.469329 0.472222 T= 0.353781 0.353009 0.352431   0.176183 0.176553 0.176505 T 5 = 0.470743 0.47039 0.470775 0.353074 0.353057 0.35272
34. 34.   0.17554 0.177662 0.175347 4 0.470679 0.469329 0.472222 T= 0.353781 0.353009 0.352431   0.176183 0.176553 0.176505 T 5 = 0.470743 0.47039 0.470775 0.353074 0.353057 0.35272   0.176414 0.176448 0.176529 T 6 = 0.470636 0.470575 0.470583 0.35295 0.352977 0.352889 Do they converge? To what?
35. 35. A transition matrix (or corresponding Markov process) is called regular if some power of the matrix has all nonzero entries. Or, there is a positive probability of eventually moving from every state to every state.
36. 36. Theorem 2.5 If T is the transition matrix of a regular Markov process, then (a) As n → ∞, T n approaches a matrix   u1 u1 . . . u1  u2 u2 . . . u2  A= , . . . . . . . . . . . . un un . . . un all of whose columns are identical. (b) Every column u is a a probability vector all of whose components are positive.
37. 37. Theorem 2.6 If T is a regular∗ transition matrix and A and u are as above, then (a) For any probability vector x, Tn x → u as n → ∞, so that u is a steady-state vector. (b) The steady-state vector u is the unique probability vector satisfying the matrix equation Tu = u.
38. 38. Finding the steady-state vector We know the steady-state vector is unique. So we use the equation it satisﬁes to ﬁnd it: Tu = u.
39. 39. Finding the steady-state vector We know the steady-state vector is unique. So we use the equation it satisﬁes to ﬁnd it: Tu = u. This is a matrix equation if you put it in the form (T − I)u = 0
40. 40. Example (Skipping class) 0.7 0.8 If the transition matrix is T = , what is the 0.3 0.2 steady-state vector?
41. 41. Example (Skipping class) 0.7 0.8 If the transition matrix is T = , what is the 0.3 0.2 steady-state vector? Solution We can combine the equations (T − I )u = 0, u1 + u2 = 1 into a single linear system with augmented matrix     −3/10 8/10 0 1 0 8/11  3/10 −8/10 0   0 1 3/11  1 11 00 0
42. 42. Example (Skipping class) 0.7 0.8 If the transition matrix is T = , what is the 0.3 0.2 steady-state vector? Solution We can combine the equations (T − I )u = 0, u1 + u2 = 1 into a single linear system with augmented matrix     −3/10 8/10 0 1 0 8/11  3/10 −8/10 0   0 1 3/11  1 11 00 0 8/11 So the steady-state vector is . You’ll go to class about 72% 3/11 of the time.
43. 43. Example (The Markov Dance)   1/3 1/4 0 If the transition matrix is T = 1/3 1/2 1/2, what is the 1/3 1/4 1/2 steady-state vector?
44. 44. Example (The Markov Dance)   1/3 1/4 0 If the transition matrix is T = 1/3 1/2 1/2, what is the 1/3 1/4 1/2 steady-state vector? Solution We have     −2/3 1/4 0 0 1 0 0 3/17  1/3 −1/2 1/2 0 0 1 0 8/17      1/4 −1/2  1/3 0 0 0 1 6/17  1 1 1 1 0 0 0 0