Chap 4 markov chains

1
Assoc. Prof. Ho Thanh Phong
Probability Models
International University – Dept. of ISE
 Introduction Stochastic Processes.
 Markov Chains.
 Chapman-Kolmogorov Equations
 Classification of States
 Recurrence and Transience
 Limiting Probabilities
Markov Chains4

2
Probability Models
Stochastic Processes
• A stochastic process is a collection of random variables
• Typically, T is continuous (time) and we have
• Or, T is discrete and we are observing at
discrete time points n that may or may not be evenly spaced.
• Refer to X(t) as the state of the process at time t.
• The state space of the stochastic process is the set of all possible
values of X(t): this set may be discrete or continuous as well.
  ,X t t T
 , 0,1,2,...nX n 
  , 0X t t 

3
Probability Models
Markov Chains
• A Markov chain is a stochastic process , where
each Xn belongs to the same subset of {0, 1, 2, …}, and
for all states i0, i1,…, in-1 and all n  0 .
• Denote as the transition probability
Then
Let be the matrix of one-step transition probabilities.
 , 0,1,2,...nX n 
   1 1 1 1 1 0 0 1, ,..., ,n n n n n nP X j X i X i X i X i P X j X i          
 1ij n nP P X j X i  
ijP   P
nn XX statepresenton theonlydepends1
1,anyFor
,allfor0


jall
ij
ij
Pi
jiP

4
Probability Models
Markov Chains - Examples
Example 1: Forecasting the weather
Xn : weather of day n
S = {0 : rain , 1 : no rain}
P00 =  , P10 = 











1
1
P

5
Probability Models
Example 2: Forecasting the weather
• If it rains for the past two days then it will rain tomorrow with probability
0.7
• If it rains today but not yesterday then it will rain tomorrow with
probability 0.5
• If it rains yesterday but not today then it will rain tomorrow with
probability 0.4
• If it has not rain in the past two days then it will rain tomorrow with
probability 0.2
States: 0: RR, 1: NR, 2: RN, 3: NN













8.02.0
6.04.0
05.05.
03.07.
P

6
Probability Models
Example 3: Random Walks
• State Space (S): 0, ±1, ±2, ±3, ±4,….
• Pi, i + 1 = p ; Pi, i - 1 = 1 – p i = 0, 1, …
• At each point of time, either it takes one step to the right with
probability p, or one step to the left with probability 1-p.
S
-2 -1 0 1 2 ……

7
Probability Models
Example 3: A Gambling Model
Gambler quits if he goes broke or if he obtains a fortune N.
 


 p
p
1with$1loses
with$1wins
playeachatGambler
statesabsorbingareNand0:1
1,...,3,2,11;
00
1,1,

 
NN
iiii
PP
NipPpP
0 1 2 i-1 i i+1 N-1 N
1 1
p p p p
q q q q

8
Probability Models
Chapman-Kolmogorov Equations
• n-step Transition Probabilities:
• Chapman-Kolmogorov Equations
• Let P(n) be the matrix of n-step transition probabilities:
• So, proven by induction,
{ | }, , 0, , 0n
ij n m mP P X j X i n m i j    
0
, , 0, , 0n m n m
ij ik kj
k
P P P n m i j



  
     n m n m
P P P
 n n
P P

9
Probability Models
Example
• Transition probability matrix: 






6.4.
3.7.
P
With: i = 1: it rains; i = 2: it does not rain







4332.5668.
4251.5749.4
P
• If: Prob. it rains today is α1 = 0.4
Prob. it does not rain today is α2 = 0.6
   43.57.
4332.5668.
4251.5749.
6.4.4






PThen 

10
Probability Models
Classification of States
• State j is accessible from state i if
• If j is accessible from i and i is accessible from j, we say that states i
and j communicate (i  j).
• Communication is a class property:
(i) State i communicates with itself, for all i  0
(ii) If i  j then j  i : communicate is commutative
(iii) If i  j and j  k, then i  k : communicate is transitive
• Therefore, communication divides the state space up into mutually
exclusive classes.
• If all the states communicate, the Markov chain is irreducible.
0 for some 0n
ijP n 

11
Probability Models
Classification of States
An irreducible Markov chain:
0
3 4
21
An reducible Markov chain:
0
3 4
21

12
Probability Models
Recurrence vs. Transience
• Let fi be the probability that, starting in state i, the process will ever
reenter state i. If fi = 1, the state is recurrent, otherwise it is transient.
– If state i is recurrent then, starting from state i, the process will reenter state i
infinitely often (w/prob. 1).
– If state i is transient then, starting in state i, the number of periods in which the
process is in state i has a geometric distribution with parameter 1 – fi.
• state i is recurrent if and transient if
• Recurrence (transience) is a class property: If i is recurrent
(transient) and i  j then j is recurrent (transient).
• A special case of a recurrent state is if Pii = 1 then i is absorbing.
1
n
iin
P


  1
n
iin
P


 

13
Probability Models
Recurrence vs. Transience (2)
• Not all states in a finite Markov chain can be transient.
• All states of a finite irreducible Markov chain are recurrent.
• If state i is recurrent and state i does not communicate with
state j, then
– when a process enters a recurrent class of states it can
never leave that class.
– A recurrent class is often called a closed class
0ijP

14
Probability Models
Examples
recurrentarestatesAll
0010
0010
0001
5.5.00












P
     transientrecurrentP 
















 4:Class;3,2,1,0:Classes
5.0025.25.
05.5.00
05.5.00
0005.5.
0005.5.
recurrentarestatesalli
0100
005.5.
1000
1000













 rreducibleP

15
Probability Models
Limiting Probabilities
• If whenever n is not divisible by d, and d is the largest
integer with this property, then state i is periodic with period d.
• If a state has period d = 1, then it is aperiodic.
• If state i is recurrent and if, starting in state i, the expected time until
the process returns to state i is finite, it is positive recurrent
(otherwise it is null recurrent).
• A positive recurrent, aperiodic state is called ergodic.
0n
iiP 

16
Probability Models
Limiting Probabilities (2)
Theorem:
• For an irreducible ergodic Markov chain, exists for all
j and is independent of i.
• Furthermore, pj is the unique nonnegative solution of
• The probability pj also equals the long run proportion of time that the
process is in state j.
lim n
j ij
n
Pp


0
0
, 0
1
j i ij
i
j
j
P jp p
p




 




17
Probability Models
Limiting Probabilities – Examples











1
1
P
   


p


p
pp
ppp
ppp













1
1
;
1
1
11
:iesprobabilitLimiting
10
10
101
100

18
Probability Models
Limiting Probabilities (3)
• The long run proportions pj are also called stationary probabilities
because if then
• Let mjj be the expected number of transitions until the Markov chain,
starting in state j, returns to state j (finite if state j is positive
recurrent). Then
 0 jP X j p 
  for all , 0n jP X j n jp  
1jj jm p

19
Probability Models
Application: Gambler’s Ruin Problem
• Gambler at each play of the game has prob. p to win one unit and has
prob. q=1-p of losing one unit. Successive plays are independent.
• What is the probability that, starting with i units, the gambler’s
fortune will reach N before going broke?
• Let Xn = player’s fortune at time n:
{Xn ; n = 0,1,2…} is a Markov chain with transition probabilities:





 1,...,2,11
1
1,1,
00
NipPP
PP
iiii
NN
• This Markov chain has three classes:
– {0} and {N} - Recurrent
– {1,2,…,N-1} - Transient

20
Probability Models
Application: Gambler’s Ruin Problem (2)
• Let Pi ; 0,1,2,...,N : Prob., starting with i, the gambler reaches N.
• Conditioning on the next game, we have:
  1,...,2,11111   NiforPP
p
q
PPorqPpPP iiiiiii
• Note that: P0 = 0
 Ni
p
qifiP
p
qifP
p
q
p
q
P
i
i ,...,2
1
1
1
1
1
1



















21
Probability Models
• Moreover, PN = 1


















2
11
2
1
1
1
1
pif
N
pif
p
q
p
q
P
N
 Ni
pif
N
i
pif
p
q
p
q
P
N
i
i ,...,2,1,0
2
1
2
1
1
1
























22
Probability Models
• For N → ∞:












2
10
2
11
pif
pif
p
q
P
i
i
• For p > 1/2: there is a positive prob. that the gambler’s fortune will
increase indefinitely.
• For p ≤ 1/2: the gambler will “almost certainly” go broke against an
infinitely rich adversary.

23
Probability Models
Mean First Passage Time of Recurrent States
 
1
1


k
k
ijf
• For an ergodic Markov chain, it is possible to go from one state to
another state in a finite number of transitions. Hence:
fij
(k): Prob. of going from i to j for the first time in exactly k transitions.
• Mean first passage time:  




1k
k
ijij kf
• Mean first passage time can be found by solving:




jkk
kjikij P
,0
1 

24
Probability Models
Example
0,1,2statesfrom3statetotimepassagefirstmeantheFind
0001
25.5.25.0
0100
05.5.0












P
5;6;5.6
5.025.001
1001
5.05.001
231303
23130323
23130313
23130303














Chap 4 markov chains

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Chap 4 markov chains

Similar to Chap 4 markov chains (20)

Recently uploaded

Recently uploaded (20)

Chap 4 markov chains