Upcoming SlideShare
×

# Markov chains1

3,372 views
3,118 views

Published on

2 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
3,372
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
169
0
Likes
2
Embeds 0
No embeds

No notes for slide

### Markov chains1

1. 1. Markov Chains
2. 2. Description Sometimes we are interested in how a random variable changes over time. The study of how a random variable evolves over time includes Stochastic Processes .
3. 3. What is a Stochastic Process? <ul><li>Suppose we observe some characteristic of a system at discrete points in time. </li></ul><ul><li>Let X t be the value of the system characteristic at time t . In most situations, X t is not known with certainty before time t and may be viewed as a random variable. </li></ul><ul><li>A discrete-time stochastic process is simply a description of the relation between the random variables X 0 , X 1 , X 2 ….. </li></ul>
4. 4. <ul><li>A continuous –time stochastic process is simply the stochastic process in which the state of the system can be viewed at any time, not just at discrete instants in time. </li></ul><ul><li>For example, the number of people in a supermarket t minutes after the store opens for business may be viewed as a continuous-time stochastic process. </li></ul>
5. 5. The Gambler’s Ruin Problem <ul><li>At time 0, I have Rs. 2. At times 1, 2, …, I play a game in which I bet Rs. 1, with probabilities p, I win the game, and with probability 1 – p, I lose the game. My goal is to increase my capital to Rs. 4, and as soon as I do, the game is over. The game is also over if my capital is reduced to 0. </li></ul><ul><ul><li>Let X t represent my capital position after the time t game (if any) is played </li></ul></ul><ul><ul><li>X 0 , X 1 , X 2 , …. May be viewed as a discrete-time stochastic process </li></ul></ul>
6. 6. What is a Markov Chain? <ul><li>One special type of discrete-time stochastic process is called a Markov Chain. </li></ul><ul><li>Definition: A discrete-time stochastic process is a Markov chain if, for t = 0,1,2… and all states P ( X t+1 = i t+1 | X t = i t , X t-1 = i t-1 ,…, X 1 = i 1 , X 0 = i 0 ) = P ( X t+1 = i t+1 | X t = i t ) </li></ul><ul><li>Essentially this says that the probability distribution of the state at time t +1 depends on the state at time t ( i t ) and does not depend on the states the chain passed through on the way to i t at time t . </li></ul>
7. 7. <ul><li>In our study of Markov chains, we make further assumption that for all states i and j and all t , </li></ul><ul><li>P ( X t+ 1 = j| X t = i ) is independent of t . </li></ul><ul><li>This assumption allows us to write </li></ul><ul><ul><li>P ( X t+1 = j| X t = i ) = p ij where p ij is the probability that given the system is in state i at time t , it will be in a state j at time t +1. </li></ul></ul><ul><li>If the system moves from state i during one period to state j during the next period, we that a transition from i to j has occurred. </li></ul>
8. 8. <ul><li>The p ij ’s are often referred to as the transition probabilities for the Markov chain. </li></ul><ul><li>This equation implies that the probability law relating the next period’s state to the current state does not change over time. </li></ul><ul><li>It is often called the Stationary Assumption and any Markov chain that satisfies it is called a stationary Markov chain . </li></ul><ul><li>We also must define q i to be the probability that the chain is in state i at the time 0; in other words, P ( X 0 = i ) = q i . </li></ul>
9. 9. <ul><li>We call the vector q = [ q 1 , q 2 ,…q s ] the initial probability distribution for the Markov chain. </li></ul><ul><li>In most applications, the transition probabilities are displayed as an s x s transition probability matrix P . The transition probability matrix P may be written as </li></ul>
10. 10. TPM: The Gambler’s Ruin Problem <ul><li>Rs. 0 Rs. 1 Rs. 2 Rs. 3 Rs. 4 </li></ul>
11. 11. <ul><li>For each i </li></ul><ul><li>We also know that each entry in the P matrix must be nonnegative. </li></ul><ul><li>Hence, all entries in the transition probability matrix are nonnegative, and the entries in each row must sum to 1. </li></ul>
12. 12. Question <ul><li>A company has two machines. During any day, each machine that is working at the beginning of the day has a 1/3 chance of breaking down. If a machine breaks down during the day it is sent to a repair facility and will be working two days after it breaks down. (Thus, if a machine breaks down during day-3, it will be working at the beginning of day 5) </li></ul><ul><li>Letting the state of the system to be the number of machines working at the beginning of the day, formulate a transition probability matrix for this situation. </li></ul>
13. 13. n -Step Transition Probabilities <ul><li>A question of interest when studying a Markov chain is: If a Markov chain is in a state i at time m , what is the probability that n periods later than the Markov chain will be in state j ? </li></ul><ul><li>This probability will be independent of m , so we may write P ( X m+n = j | X m = i ) = P ( X n = j | X 0 = i ) = P ij ( n ) where P ij ( n ) is called the n -step probability of a transition from state i to state j . </li></ul><ul><li>For n > 1, P ij ( n ) = ij th element of P n </li></ul>
14. 14. The Cola Example <ul><li>Suppose the entire cola industry produces only two colas. </li></ul><ul><li>Given that a person last purchased cola 1, there is a 90% chance that her next purchase will be cola 1. </li></ul><ul><li>Given that a person last purchased cola 2, there is an 80% chance that her next purchase will be cola 2. </li></ul><ul><ul><ul><li>If a person is currently a cola 2 purchaser, what is the probability that she will purchase cola 1 two purchases from now? </li></ul></ul></ul><ul><ul><ul><li>If a person is currently a cola 1 purchaser, what is the probability that she will purchase cola 1 three purchases from now? </li></ul></ul></ul>
15. 15. The Cola Example <ul><li>We view each person’s purchases as a Markov chain with the state at any given time being the type of cola the person last purchased. </li></ul><ul><li>Hence, each person’s cola purchases may be represented by a two-state Markov chain, where </li></ul><ul><ul><li>State 1 = person has last purchased cola 1 </li></ul></ul><ul><ul><li>State 2 = person has last purchased cola 2 </li></ul></ul><ul><li>If we define X n to be the type of cola purchased by a person on her n th future cola purchase, then X 0 , X 1 , … may be described as the Markov chain with the following transition matrix: </li></ul>
16. 16. The Cola Example <ul><li>We can now answer questions 1 and 2. </li></ul><ul><li>We seek P ( X 2 = 1| X 0 = 2) = P 21 (2) = element 21 of P 2 : </li></ul>
17. 17. The Cola Example <ul><ul><li>Hence, P 21 (2) =.34. This means that the probability is .34 that two purchases in the future a cola 2 drinker will purchase cola 1. </li></ul></ul><ul><ul><li>By using basic probability theory, we may obtain this answer in a different way. </li></ul></ul><ul><li>We seek P 11 (3) = element 11 of P 3 : </li></ul><ul><li>Therefore, P 11 (3) = .781 </li></ul>
18. 18. <ul><li>Many times we do not know the state of the Markov chain at time 0. Then we can determine the probability that the system is in state i at time n by using the reasoning. </li></ul><ul><li>Probability of being in state j at time n where q =[q 1 , q 2 , … q 3 ]. </li></ul><ul><li>(This is an unconditional probability). </li></ul>
19. 19. Limiting probabilities <ul><li>To illustrate the behavior of the n -step transition probabilities for large values of n , we have computed several of the n -step transition probabilities for the Cola example. </li></ul><ul><li>This means that for large n, no matter what the initial state, there is a .67 chance that a person will be a cola 1 purchaser. </li></ul><ul><li>We can easily multiply matrices on a spreadsheet using the MMULT command. </li></ul>
20. 20. Question Find the equilibrium market shares of two firms whose probability transition matrix is as follows .5 .5 B .3 .7 A B A
21. 21. Example <ul><li>Suppose we have a Markov transition matrix: </li></ul>1 2 3 1/2 1/2 1/2 1/2 1
22. 22. All states communicate with each other Starting from 1, the MC can return to 1 in three steps via two possible routes: Route 1: 1 to 3 to 2 to 1 with probability .5 ×1 ×.5 = 1/4 Route 2: 1 to 2 to 2 to 1 with probability .5 ×.5×.5 = 1/8 Hence the required probability is =1/4+1/8 = 3/8 .
23. 23. Steady-State Probabilities <ul><li>Steady-state probabilities are used to describe the long-run behavior of a Markov chain. </li></ul><ul><li>Theorem 1: Let P be the transition matrix for an s -state ergodic chain. Then there exists a vector π = [ π 1 π 2 … π s ] such that </li></ul>
24. 24. <ul><li>Theorem 1 tells us that for any initial state i , </li></ul><ul><li>The vector π = [ π 1 π 2 … π s ] is often called the steady-state distribution , or equilibrium distribution , for the Markov chain. </li></ul>
25. 25. An Example <ul><li>A supermarket stocks 3 brands of coffee, A , B , and C, and it has been observed that customers switch from brand to brand according to the following transition matrix: </li></ul>In the long In the long run, what fraction of the customers purchase the respective brands?
26. 26. Solution <ul><li>Since the chain is ergodic (all states are communicating, each state is recurrent and aperiodic), the steady-state distribution exists. </li></ul><ul><li>Solving  =  P gives </li></ul><ul><li> 1 =(3/4)  1 +(1/4)  3 </li></ul><ul><li> 2 =(1/4)  1 +(2/3)  2 +(1/4)  3 </li></ul><ul><li> 3 =(1/3)  2 +(1/2)  3 </li></ul><ul><li>Subject to  1 +  2 +  3 =1. Solving the equations gives </li></ul><ul><li> 1 =(2/7),  2 =(3/7),  3 =2/7. </li></ul>
27. 27. Inventory Example <ul><li>A camera store stocks a particular model camera that can be ordered weekly. Let D 1 , D 2 , … represent the demand for this camera (the number of units that would be sold if the inventory is not depleted) during the first week, second week, …, respectively. It is assumed that the D i ’s are independent and identically distributed random variables having a Poisson distribution with a mean of 1. Let X 0 represent the number of cameras on hand at the outset, X 1 the number of cameras on hand at the end of week 1, X 2 the number of cameras on hand at the end of week 2, and so on. </li></ul><ul><ul><li>Assume that X 0 = 3. </li></ul></ul><ul><ul><li>On Saturday night the store places an order that is delivered in time for the next opening of the store on Monday. </li></ul></ul><ul><ul><li>The store using the following order policy: If there are no cameras in stock, 3 cameras are ordered. Otherwise, no order is placed. </li></ul></ul><ul><ul><li>Sales are lost when demand exceeds the inventory on hand </li></ul></ul>
28. 28. Inventory Example <ul><li>X t is the number of Cameras in stock at the end of week t (as defined earlier), where X t represents the state of the system at time t </li></ul><ul><li>Given that X t = i, X t+1 depends only on D t+1 and X t (Markovian property) </li></ul><ul><li>D t has a Poisson distribution with mean equal to one. This means that P(D t+1 = n) = e -1 1 n /n! for n = 0, 1, … </li></ul><ul><li>P(D t = 0 ) = e -1 = 0.368 </li></ul><ul><li>P(D t = 1 ) = e -1 = 0.368 </li></ul><ul><li>P(D t = 2 ) = (1/2)e -1 = 0.184 </li></ul><ul><li>P(D t  3 ) = 1 – P(D t  2) = 1 – (.368 + .368 + .184) = 0.08 </li></ul><ul><li>X t+1 = max(3-D t+1 , 0) if X t = 0 and X t+1 = max(X t – D t+1 , 0) if X t  1, for t = </li></ul><ul><li>0, 1, 2, …. </li></ul>
29. 29. Inventory Example: (One-Step) Transition Matrix <ul><li>P 03 = P(D t+1 = 0) = 0.368 </li></ul><ul><li>P 02 = P(D t+1 = 1) = 0.368 </li></ul><ul><li>P 01 = P(D t+1 = 2) = 0.184 </li></ul><ul><li>P 00 = P(D t+1  3) = 0.080 </li></ul>
30. 30. Inventory Example: Transition Diagram 0 1 2 3
31. 31. Inventory Example: (One-Step) Transition Matrix
32. 32. Transition Matrix: Two-Step <ul><li>P (2) = PP </li></ul>
33. 33. Transition Matrix: Four-Step <ul><li>P (4) = P (2) P (2) </li></ul>
34. 34. Transition Matrix: Eight-Step <ul><li>P (8) = P (4) P (4) </li></ul>
35. 35. Steady-State Probabilities <ul><li>The steady-state probabilities uniquely satisfy the following steady-state equations </li></ul><ul><li> 0 =  0 p 00 +  1 p 10 +  2 p 20 +  3 p 30 </li></ul><ul><li> 1 =  0 p 01 +  1 p 11 +  2 p 21 +  3 p 31 </li></ul><ul><li> 2 =  0 p 02 +  1 p 12 +  2 p 22 +  3 p 32 </li></ul><ul><li> 3 =  0 p 03 +  1 p 13 +  2 p 23 +  3 p 33 </li></ul><ul><li>1 =  0 +  1 +  2 +  3 </li></ul>
36. 36. Steady-State Probabilities: Inventory Example <ul><li> 0 = .080  0 + .632  1 + .264  2 + .080  3 </li></ul><ul><li> 1 = .184  0 + .368  1 + .368  2 + .184  3 </li></ul><ul><li> 2 = .368  0 + .368  2 + .368  3 </li></ul><ul><li> 3 = .368  0 + .368  3 </li></ul><ul><li>1 =  0 +  1 +  2 +  3 </li></ul><ul><li> 0 = .286,  1 = .285,  2 = .263,  3 = .166 </li></ul><ul><li>The numbers in each row of matrix P (8) match the corresponding steady-state probability </li></ul>
37. 37. Mean First Passage Times <ul><li>For an ergodic chain, let m ij = expected number of transitions before we first reach state j , given that we are currently in state i; m ij is called the mean first passage time from state i to state j . </li></ul><ul><li>In the example, we assume we are currently in state i . Then with probability p ij , it will take one transition to go from state i to state j . For k ≠ j , we next go with probability p ik to state k . In this case, it will take an average of 1 + m kj transitions to go from i and j . </li></ul>
38. 38. <ul><li>This reasoning implies </li></ul><ul><li>By solving the linear equations of the equation above, we find all the mean first passage times. It can be shown that </li></ul>
39. 39. <ul><li>For the cola example, π 1 =2/3 and π 2 = 1/3 </li></ul><ul><li>Hence, m 11 = 1.5 and m 22 = 3 </li></ul><ul><li>m 12 = 1 + p 11 m 12 = 1 + .9m 12 </li></ul><ul><li>m 21 = 1 + p 22 m 21 = 1 + .8m 21 </li></ul><ul><li>Solving these two equations yields, </li></ul><ul><li>m 12 = 10 and m 21 = 5 </li></ul>