Discrete time Markov Chains

1. Discrete-Time Markov Models
Introduction to Stochastic Processes
Bernardo D’Auria
SCQ0093538 2022/23 1st Semester

2. Markov Property
Definition 1.1 Let 𝑋𝑛 𝑛≥0 be a discrete-time stochastic process with a
countable state space 𝐸.
If ∀𝑛 ≥ 0 and states 𝑖0, 𝑖1, … , 𝑖𝑛−1, 𝑖, 𝑗 ∈ 𝐸
ℙ 𝑋𝑛+1 = 𝑗|𝑋𝑛 = 𝑖, 𝑋𝑛−1 = 𝑖𝑛−1, … 𝑋0 = 𝑖0 = ℙ 𝑋𝑛+1 = 𝑗|𝑋𝑛 = 𝑖
whenever both sides are well-defined, then the process 𝑋𝑛 𝑛≥0 is
called a discrete-time Markov chain.
In addition, if ℙ 𝑋𝑛+1 = 𝑗|𝑋𝑛 = 𝑖 = 𝑝𝑖𝑗, that is independently of 𝑛, the
chain is called homogenous, (HMC).
In this case we define the matrix P = 𝑝𝑖𝑗 as the transition matrix.
2 of 26

3. Stochastic matrix
Definition. A matrix 𝑀 = 𝑚𝑖𝑗 is called stochastic if
𝑚𝑖𝑗 ≥ 0,
𝑗∈𝐸
𝑚𝑖𝑗 = 1
It follows that the transition matrix P is a stochastic matrix.
Even if the state space may have infinite states, the product operation of
stochastic matrix, as well as the product with a vector, are well defined
𝐶 = 𝐴 ⋅ 𝐵: 𝑐𝑖𝑗 =
𝑘∈𝐸
𝑎𝑖𝑘𝑏𝑘𝑗
𝑦𝑇
= 𝑥𝑇
𝐴: 𝑦𝑖 =
𝑘∈𝐸
𝑥𝑘𝑎𝑘𝑖
3 of 26

4. Example
Example 1.1 Machine Replacement
Let 𝑈𝑛 𝑛≥0 be a sequence of i.i.d. random variables with values
in 1,2, … , +∞
𝑈𝑛 models the lifetime of the machine 𝑛.
We assume that after any failure we replace immediately the
machine, and we denote by 𝑋𝑛 as the elapsed time the current
machine is in service.
We can show that 𝑋𝑛 𝑛≥0 is a homogeneous Markov Chain,
with non-null entries equal to
𝑝𝑖,𝑖+1 = ℙ 𝑈1 > 𝑖 + 1|𝑈1 > 𝑖
𝑝𝑖,0 = 1 − 𝑝𝑖,𝑖+1
4 of 24

5. Example
Transition diagram
where 𝑝𝑖 = 𝑝𝑖,0 = ℙ 𝑈1 = 𝑖 + 1|𝑈1 > 𝑖 .
5 of 24

6. Distribution of HMC
We define by 𝜈 the distribution of the initial state 𝑋0
𝜈 𝑖 = ℙ 𝑋0 = 𝑖
called, initial distribution.
And by 𝜈𝑛 the distribution of the state at time 𝑛, 𝑋𝑛.
Using the Bayes’s sequential rule, it follows that
ℙ 𝑋𝑛 = 𝑖, 𝑋𝑛−1 = 𝑖𝑛−1, … 𝑋0 = 𝑖0 = 𝜈 𝑖0 𝑝𝑖0𝑖1
𝑝𝑖1𝑖2
⋯ 𝑝𝑖𝑛−1𝑖𝑛
or in matrix form
𝜈𝑛
𝑇 = 𝜈𝑇𝑃𝑛
We write ℙ𝑖 ⋅ whenever 𝜈 𝑗 = 𝛿𝑖𝑗, that is when the chain starts in 𝑋0 = 𝑖.
Theorem 1.1 Distribution of an HMC
The discrete time HMC is completely characterized by the initial distribution 𝜈, and its
transition matrix 𝑃.
6 of 24

7. Filtration
Let define by ℱ𝑛 the 𝜎-algebra generated by the sets
{𝑋𝑛 = 𝑖, 𝑋𝑛−1 = 𝑖𝑛−1, … 𝑋0 = 𝑖0}
for any 𝑖0, 𝑖1, … , 𝑖𝑛 ∈ 𝐸
Then we have that ℱ𝑛 ⊂ ℱ𝑛+1.
The collection ℱ𝑛 𝑛≥0 is called filtration.
If 𝑋𝑛 𝑛≥0 is an HMC then for any A ∈ ℱ𝑛 we have
ℙ 𝑋𝑛+1 = 𝑗|𝑋𝑛 = 𝑖, 𝐴 = 𝑝𝑖𝑗
that can be written in a more succinct form as
ℙ 𝑋𝑛+1 = 𝑗|ℱ𝑛 = 𝑝𝑋𝑛𝑗
7 of 24

8. Markov Recurrences
Theorem 2.1 HMCs driven by White Noise
Let 𝑍𝑛 𝑛≥0 (the white noise), be an i.i.d sequence of random variables
with value in 𝐹 and let
f: 𝐸 × 𝐹 → 𝐸
be some function. Then with 𝑋0 ⊥ 𝑍𝑛 𝑛≥0,
𝑋𝑛+1 = f 𝑋𝑛, 𝑍𝑛
defines an HMC, with
𝑝𝑖,𝑗 = ℙ f 𝑖, 𝑍𝑛 = 𝑗
The same result follows if 𝑍𝑛+1 is conditionally independent of
𝑋0, 𝑍1, … , 𝑍𝑛−1, 𝑋1, … , 𝑋𝑛−1 given 𝑋𝑛. In this case 𝑝𝑖,𝑗 = ℙ𝑖 f 𝑖, 𝑍1 = 𝑗
8 of 24

9. Examples
Example 2.1 1-D Random Walk
𝑍𝑛~𝐵𝑒 𝑝
𝑋𝑛+1 = 𝑋𝑛 − 1 + 2𝑍𝑛
Example 2.2 Repair Shop
𝑍𝑛 𝑛≥0 i.i.d and non-negative integer-valued
𝑋𝑛+1 = 𝑋𝑛 − 1 +
+ 𝑍𝑛+1
9 of 24

10. Examples
Example 2.3 Inventory with (s,S)-strategy
𝑍𝑛 𝑛≥0 i.i.d non-negative integer-valued
𝑋𝑛+1 =
𝑋𝑛 − 𝑍𝑛+1
+ if 𝑠 ≤ 𝑋𝑛 ≤ 𝑆
𝑆 − 𝑍𝑛+1
+
if 𝑋𝑛 < 𝑠
Example 2.4 Branching process (Galton-Watson process)
𝑍𝑛 𝑛≥0 i.i.d with 𝑍𝑛 = 𝑍𝑛
1
, 𝑍𝑛
2
, … ,
𝑍𝑛
𝑘
𝑘≥1
i.i.d. and non-negative integer valued
𝑋𝑛+1 =
𝑘=1
𝑋𝑘
𝑍𝑛+1
(𝑘)
10 of 24

11. First-Step Analysis
Let us anticipate that a closes set 𝐴 ⊂ 𝐸 is closed if
𝑗∈𝐴
𝑝𝑖𝑗 = 1, ∀𝑖 ∈ 𝐴
First step analysis is a simple but powerful technique to
compute many properties, such as for example
absorption probability in closed sets.
Example 3.1 & 3.4 Gambler’s Ruin
Example 2 Cat Eats Mouse Eat Cheese
11 of 24

12. Hitting probabilities
Definition Let 𝑇𝐴 𝜔 = inf 𝑛 ≥ 0: 𝑋𝑛 𝜔 ∈ 𝐴 be the hitting time of a set
𝐴 ⊂ 𝐸, we define the hitting probability of 𝐴 starting in 𝑖 ∈ 𝐸, as
𝑢𝑖
𝐴
= ℙ𝑖 𝑇𝐴 < ∞
[N’97] Theorem 1.3.2. The vector 𝑢𝐴 = 𝑢𝑖
𝐴
: 𝑖 ∈ 𝐸 of hitting probabilities is
the minimal non-negative solution to the system of linear equations
𝑢𝑖
𝐴
= 1 for 𝑖 ∈ 𝐴
𝑢𝑖
𝐴
=
𝑗∈𝐸
𝑝𝑖𝑗𝑢𝑗
𝐴
for 𝑖 ∉ 𝐴
(Minimality means that if 𝑥 = 𝑥𝑖: 𝑖 ∈ 𝐸 is another solution with
𝑥𝑖 ≥ 0 for all 𝑖 ∈ 𝐸, then 𝑥𝑖 ≥ 𝑢𝑖 hi all 𝑖 ∈ 𝐸.)
[N’97] Example 1.3.1
12 of 24
1 2 3 4
1 2
1 2
1 2
1 2

13. Mean hitting times
Definition Let 𝑇𝐴 𝜔 = inf 𝑛 ≥ 0: 𝑋𝑛 𝜔 ∈ 𝐴 be the hitting time of a set
𝐴 ⊂ 𝐸, we define the mean hitting probability of 𝐴 starting in 𝑖 ∈ 𝐸,
𝑚𝑖
𝐴
= 𝔼𝑖 𝑇𝐴
[N’97] Theorem 1.3.5. The vector of mean hitting times
𝑚𝐴 = 𝑚𝑖
𝐴
: 𝑖 ∈ 𝐸
is the minimal non-negative solution to the system of linear equations
𝑚𝑖
𝐴
= 0 for 𝑖 ∈ 𝐴
𝑚𝑖
𝐴
= 1 +
𝑗∈𝐸
𝑝𝑖𝑗𝑚𝑗
𝐴
for 𝑖 ∉ 𝐴
We write 𝑇𝑖 for 𝑇 𝑖
13 of 24

14. Exercises
Exercise. Given the transition matrix
𝑃 =
0 1
2 0 1
2 0
1
2 0 1
2 0 0
0 0 1 0 0
1
3 0 1
3 0 1
3
0 0 0 0 1
• Draw transition graph of the chain.
• Compute the absorption probability of the set 5 starting
from 1.
14 of 24

15. Exercises
Exercise. Consider the chain
• Compute the mean absorption time to 𝑐 starting
from a.
15 of 24
a c
𝑏1
𝑛 + 1 −1
1
1
𝑏𝑛
1
𝑛 + 1 −1
𝑛 + 1 −1
𝑛 states

16. Topology of the Transition Matrix
Definition 4.1 Communication
Let 𝑗, 𝑖 ∈ 𝐸, if ∃𝑛 ≥ 0 such that 𝑝𝑖𝑗
(𝑛)
> 0, then we say that 𝑗
is accessible from 𝑖, and we write 𝑖 → 𝑗.
If 𝑖 → 𝑗 and 𝑗 → 𝑖 then we say 𝑖 and 𝑗 communicate, and we
write 𝑗 ↔ 𝑖
Properties of the relation of Communication
• 𝑖 ↔ 𝑖 (reflexivity)
• 𝑖 ↔ 𝑗 ⇒ 𝑗 ↔ 𝑖 (symmetry)
• 𝑖 ↔ 𝑗, 𝑗 ↔ 𝑘 ⇒ 𝑖 ↔ 𝑘 (transitivity)
16 of 24

17. Topology of the Transition Matrix
Definition 4.2 Closed sets
A state 𝑖 ∈ 𝐸 is closed if 𝑝𝑖𝑖 = 1.
A set 𝐶 ⊂ 𝐸 is closed if 𝑗∈𝐶 𝑝𝑖𝑗 = 1, ∀𝑖 ∈ 𝐶
Example 4.1
Definition 4.3 Irreducibility
An HMC is irreducible if it has only one communication class.
17 of 24
1
2
3
4
6
5
8
7

18. Topology of the Transition Matrix
Definition 4.4 Arithmetic definition of Period
Let 𝑖 ∈ 𝐸 be a state and consider the set
𝐷𝑖 = 𝑛 ∈ ℕ: 𝑝𝑖𝑖 > 0
If 𝐷𝑖 = ∅, we set the period 𝑑𝑖 = ∞ otherwise
𝑑𝑖 = GCD 𝐷𝑖
If 𝑑𝑖 = 1 we call 𝑖 aperiodic.
Theorem 4.2 Period is a Class property
If 𝑖 ↔ 𝑗 then 𝑑𝑖 = 𝑑𝑗.
18 of 24

19. Topology of the Transition Matrix
Theorem 4.1 Cyclic Structure
Any irreducible HMC has a unique partition of 𝐸 into 𝑑 classes,
𝐶0, 𝐶1, … , 𝐶𝑑−1 such that ∀𝑘, 𝑖 ∈ 𝐶𝑘
𝑗∈𝐶𝑘+1
𝑝𝑖𝑗 = 1
where 𝑑 is maximal and it is equal to 𝑑𝑖, ∀𝑖 ∈ 𝐸.
19 of 24

20. Topology of the Transition Matrix
Example 4.2
HMC with period 𝑑 = 3.
The general structure of P with period 𝑑 = 4 is the following
P4𝑛+1
=
0 𝐴0 0 0
0 0 𝐴1 0
0 0 0 𝐴2
𝐴3 0 0 0
, P4𝑛+2
=
0 0 𝐵0 0
0 0 0 𝐵1
𝐵2 0 0 0
0 𝐵4 0 0
,
P4𝑛+3
=
0 0 0 𝐶0
𝐶1 0 0 0
0 𝐶2 0 0
0 0 𝐶3 0
, P4𝑛+4
=
𝐷0 0 0 0
0 𝐷1 0 0
0 0 𝐷2 0
0 0 0 𝐷3
20 of 24
1
2
3 4
6
5
7

21. Steady state
Definition 5.1 Stationary Distribution
A probability distribution 𝜋 satisfying
𝜋𝑇
= 𝜋𝑇
⋅ 𝑃, global balance equations
is called a stationary distribution of the transition matrix 𝑃, or of the
corresponding HMC.
The global balance equation implies that for all states 𝑖 ∈ E,
𝜋𝑖 =
𝑗∈𝐸
𝜋𝑗 𝑝𝑗𝑖
and iterating, we get
𝜋𝑇 = 𝜋𝑇 ⋅ 𝑃𝑛
Theorem 5.1 Steady State
An HMC started whit a stationary distribution is stationary.
21 of 24

22. Examples
Example 5.1
𝐸 = 1,2 and 𝑃 =
1 − 𝛼 𝛼
𝛽 1 − 𝛽
where
𝛼, 𝛽 ∈ 0,1 . Find the stationary distribution.
Example 5.4
It may exist more than one stationary distribution
Example 5.3 Symmetric Random Walk
Example 5.5 Repair Shop
22 of 24

23. Global Balance Equations and Probability Flows
Global balance equation
Consider the component 𝑖 of 𝜋𝑇 = 𝜋𝑇 ⋅ 𝑃
𝜋 𝑖 =
𝑗∈𝐸
𝜋 𝑗 𝑝𝑗𝑖
subtracting 𝜋 𝑖 𝑝𝑖𝑖 from both terms we have
𝑗∈𝐸,𝑗≠𝑖
𝜋 𝑖 𝑝𝑖𝑗 =
𝑗∈𝐸,𝑗≠𝑖
𝜋 𝑗 𝑝𝑗𝑖
that has a graphical interpretation in terms of
probability flows
23 of 24
𝑖

24. Detailed Balance Equations and Probability Flows
Detailed balance equation
If a distribution 𝜋 satisties for all 𝑖, 𝑗 ∈ 𝐸,
𝜋 𝑖 𝑝𝑖𝑗 = 𝜋 𝑗 𝑝𝑗𝑖, detailed balance equations
then 𝜋 is a stationary distribution.
24 of 24
𝑖 𝑗

25. Time reversal
Transition matrix of the reversed chain
Take the stationary distribution 𝜋 and define the
stochastic matrix 𝑄 = (𝑞𝑖𝑗) with entries
𝑞𝑖𝑗 = 𝑝𝑗𝑖
𝜋 𝑗
𝜋 𝑖
then 𝑄 is the transition matrix of the stationary
time-reversed chain
𝑞𝑖𝑗 = ℙ𝜋 𝑋−1 = 𝑗|𝑋0 = 𝑖
25 of 24

26. Time reversal
Theorem 6.1 Reversal test
If 𝑃and 𝑄 are stochastic matrices and 𝜋 is a distribution
satisfying 𝜋 𝑖 𝑞𝑖𝑗 = 𝜋 𝑗 𝑝𝑖𝑗, ∀𝑖, 𝑗 ∈ 𝐸, then 𝜋 is
stationary.
Example 6.2 & 5.2 The urn of Ehrenfest
Example 6.3 Random walk on graphs
Example 6.4 Birth and Death process
26 of 24

27. Strong Markov property
Definition 7.1 Stopping time
A random time 𝜏 ∈ ℕ0 ∪ +∞ is a stopping time with
respect to a stochastic process 𝑋𝑛 𝑛≥0 if the event
𝜏 = 𝑛 ∈ ℱ𝑛
that is, there is a function fn with values in 0,1 such
that
૤ 𝜏=𝑛 = f𝑛 𝑋0, 𝑋1, … , 𝑋𝑛−1, 𝑋𝑛
Example 7.1 & 7.5 Return times & successive return times
𝑅𝑖 = inf 𝑛 ≥ 1: 𝑋𝑛 = 𝑖
note that 𝑅𝑖 ≠ 𝑇𝑖.
27 of 24

28. Strong Markov property
Theorem 7.1 Strong Markov property
Let 𝑋𝑛 𝑛≥0 be an HMC with countable state space 𝐸
and transition matrix 𝑃, and 𝜏 a stopping time with
respect to it. Then for any state 𝑖 ∈ 𝐸, given that 𝑋𝜏 = 𝑖
(i.e., 𝜏 < ∞) the following holds
• The process after 𝜏 and the process before 𝜏 are
independent
• The process after 𝜏 is an HMC with transition matrix 𝑃
Exercise 2.7.2 Markov chain restricted to a subset of 𝐸
28 of 24

29. Regeneration
Let
𝑁𝑖 =
𝑛≥1
૤ 𝑋𝑛=𝑖
be the number of visits to 𝑖 ∈ 𝐸
Theorem 7.2 Visits to a state
The distribution of 𝑁𝑖, given 𝑋0 = 𝑗, is
ℙ𝑗 𝑁𝑖 = 𝑟 =
𝑓𝑗𝑖𝑓𝑖𝑖
𝑟−1
1 − 𝑓𝑖𝑖 for 𝑟 ≥ 1
1 − 𝑓𝑗𝑖 for 𝑟 = 0
where 𝑓𝑗𝑖 = ℙ𝑗 𝑅𝑖 < ∞ , with 𝑅𝑖 the return time to 𝑖.
29 of 24

30. Regeneration
Theorem 7.3 Recurrence
For any 𝑖 ∈ 𝐸
ℙ𝑖 𝑅𝑖 < ∞ = 1 ⇔ ℙ𝑖 𝑁𝑖 = ∞ = 1
and
ℙ𝑖 𝑅𝑖 < ∞ < 1 ⇔ ℙ𝑖 𝑁𝑖 = ∞ = 0 ⇔ 𝔼𝑖 𝑁𝑖 < ∞
In particular 𝑁𝑖 = ∞ has ℙ𝑖-probability 0 or 1
30 of 24

31. Regeneration
Theorem 7.4 Regenerative Cycle Theorem
Let 𝑋𝑛 𝑛≥0 be an HMC with initial state 0 that is almost
surely visited infinitely often ℙ0 𝑁0 = ∞ = 1.
Denoting by 𝜏0 = 0, 𝜏1, 𝜏2, … the successive times to
visit 0, called regeneration times, the pieces of
trajectory, called regenerative cycles,
𝑋𝜏𝑘
, 𝑋𝜏𝑘+1, 𝑋𝜏𝑘+2, … , 𝑋𝜏𝑘+1−1 , 𝑘 ≥ 0
are independent and identically distributed.
31 of 24

32. Bibliography
[B’99] Brémaud (1999). Markov chains Gibbs fields,
Monte Carlo simulation and queues. Springer-Verlag
[N’97] Norris (1997). Markov Chains. Cambridge
University Press.
32 of 24