1. Markov Chains
Classification of States
Johar Ashfaque
A state j is said to be accessible from state i if for some n, the n-step transition probability is positive
that is to say there is positive probability of reaching state j starting from state i after some number of
time periods.
State i is called recurrent if for every state j accessible from state i, state i is also accessible from state
j. In other words, let A(i) be the set of states that are accessible from state i. Then state i is recurrent
if for all states j that belong to A(i) we have that the states i belong to A(j).
A state is called transient if it is not recurrent.
For state i being a recurrent state, the set of states A(i) forms a recurrent class.
• A Markov chain can be decomposed into one or more recurrent classes plus some possible transient
states.
• A recurrent state is accessible from all states in its class but is not accessible from recurrent states
in other classes.
• A transient state is not accessible from any recurrent state.
• At least one, possibly more, recurrent states are accessible from a given transient state.
1. We see that once a state enters a class of recurrent states it stays within the class; since all states
in the class are accessible from each other, all states in the class will be visited an infinite number
of times.
2. If the initial state is transient, then the initial portion consists of transient states and final portion
will consist of recurrent states from the same class.
Consider a recurrent class, denote it by R.
• The class is called periodic if its states can be grouped in d > 1 disjoint subsets S1, ..., Sd so that
all transitions from one subset lead to the next subset. More precisely, if
i ∈ Sk, and pij > 0 then
j ∈ Sk+1 if k = 1, ..., d − 1
j ∈ S1 if k = d
• The class is aperiodic if and only if there exists a n and a state s in the class such that
pis(n) > 0
for all i ∈ R.
1