Applied Stochastic process MTH 412 IIT Kanpur

1. Home Work 3
MTH 412
Applied Stochastic Process
1. Let Y1, Y2, . . . , be a sequence of discrete i.i.d. random variables with the following
probability mass function P(Y1 = i) = pi, where pi > 0, and
∞
i=1
pi = 1. Consider a
new sequence of random variables Xn = Y1 +. . .+Yn, for n = 1, 2, . . ., where we define
X0 = 0. Show that {Xn; n ≥ 1} is a homogeneous Markov chain. Find the transition
probability matrix. Describe the classes? Are they recurrent or transient?
2. Consider an urn with a total of D tokens; some numbered +1, and some -1. The
composition of the urn changes over time as follows. At each turn a token is picked at
random, and the sign is changed and put back in the urn. Let Xn denote the number of
+1 sign at time n. Show that Xn is a Markov chain, and find the transition probability
matrix. Is this an irreducible Markov chain?
3. Prove that for a finite Markov chain all states cannot be transient.
4. For a transient state j, prove that the series
∞
n=1
pn
jj =
fjj
1 − fjj
.
5. Prove that if C is a closed class then for all i ∈ C,
j∈C
p
(n)
ij = 1, for all n ≥ 1.
6. Prove that a closed irreducible set is minimal.
7. Consider the following transition probability matrix with the state space S = {0, 1, 2, 3, 4}








1 0 0 0 0
1
2
0 1
2
0 0
0 1
2
0 1
2
0
0 0 1
2
0 1
2
0 0 0 0 1








Find limn→∞ p
(n)
ij , for all i, j ∈ S.
8. Consider the following transition probability matrix with the state space S = {1, 2, 3, 4}





1 0 0 0
0 1 0 0
1
4
1
4
1
4
1
4
1
4
1
4
1
4
1
4





Find limn→∞ p
(n)
ij , for all i, j ∈ S. (Can you guess before solving the problem?)
1