Business Analytics CHAPTER 16 STOCHASTIC MODELS Business Analytics.PPTX

Business Analytics – The Science of Data Driven Decision Making

Stochastic Models
U Dinesh Kumar

“Problem Solving is often a matter of cooking up
an appropriate Markov chain”.
--Olle Häggström

INTRODUCTION
STOCHASTIC PROCESS

Stochastic models are powerful tools which can be
used for solving problems which are dynamic in
nature, that is, the values of the random variables
change with time.
INTRODUCTION STOCHASTIC PROCESS

Stochastic process is defined as a collection of random
variables {Xn, n  0} indexed by time (however, index can
be other than time).
• The value (cash flow) that the random variable Xn can
take is called the state of the stochastic process at
time n.
• The set of all possible values the random variable can
take is called the state space.
Stochastic Process

Homogeneous Poisson Process (HPP) is a stochastic
counting process N(t) with the following properties:
• N(0) = 0, that is the number of events by time t = 0 is zero.
• N(t) has independent increments. That is if t0 < t1 < t2 <
…< < tn, then N(t1) – N(t0), N(t2) – N(t1), …, N(tn) – N(tn1)
are independent.
• The number of events by time t, N(t), follows a Poisson
distribution, that is
Poisson Process
( )
[ ( ) ]
!
t n
e t
P N t n
n

 
 

Cumulative distribution of number of events by time t in a
Poisson process is given by
The mean, E[N(t)], and variance, Var[N(t)], of a Poisson
process N(t) are given by
E[N(t)] = t
Var[N(t)] = t
 
 
 




n
i
n
i
i
t
i
t
e
i
t
N
P
n
t
N
P
0 0 !
)
(
]
)
(
[
]
)
(
[


Poisson Process

Poisson process

Example
• Johny Sparewala (JS) is a supplier of aircraft flight control system spares
based out of Mumbai, India. The demand for hydraulic pumps used in the
flight control system follows a Poisson process. Sample data (50 cases)
on time between demands (measured in number of days) for hydraulic
pumps are shown in Table 16.1
TABLE 16.1 Time between demands (in days) for hydraulic pumps
• 104 90 45 32 12 6 30 23 58 118
• 80 12 216 71 29 188 15 88 88 94
• 63 125 108 42 77 65 18 25 30 16
• 92 114 151 10 26 182 175 189 14 11
• 83 418 21 19 73 31 175 14 226 8

Example Continued
(a) Calculate the expected number of demand for hydraulic pump spares for
next two years.
(b) Johny Sparewala would like to ensure that the demand for spares over
next two years is met in at least 90% of the cases from the spares stocked
(called fill rate) since lead time to manufacture a part is more than 2 years.
Calculate the inventory of spares that would give at least 90% fill rate.

Compound Poisson process is a stochastic process X(t)
where the arrival of events follows a Poisson process and
each arrival is associated with another independent and
identically distributed random variable Yi.
Compound Poisson process X(t) is a continuous-time
stochastic process defined as
where N(t) is a Poisson process with mean t and Yi are
independent and identically distributed random variables
with mean E(Yi) and variance Var(Yi).
Compound Poisson Process



)
(
1
)
(
t
N
k
k
Y
t
X

Example
Customers arrive at an average rate of 12 per hour to withdraw money from
an ATM and the arrivals follow a Poisson process. The money withdrawn
are independent and identically distributed with mean and variance INR
4200 and 2,50,000, respectively. If the ATM has INR 6,00,000 cash, what is
the probability that it will run out of cash in 10 hours?

Markov Chains
The condition
Is called Markov property named after the Russian
mathematician A A Markov.
If the state space S is discrete then the stochastic process
{Xn, n = 0, 1, 2, …} that satisfies the condition is called a
Markov chain
   
1 0 0 1 1 1
| , ,..., |
n n n n
P X j X i X i X i P X j X i
 
      

One-Step Transition Probabilities of
Markov Chain
Let {Xn, n = 0, 1, 2, …} be a Markov chain with state space
S. Then the conditional probability
is called the one-step transition probability. Pij gives
conditional probability of moving from state i to stage j in
one period.
   
1 0 0 1 1 1
| , ,..., |
n n n n ij
P X j X i X i X i P X j X i P
 
       

1 2 … n
1 P11 P12 … P1n
2 P21 P22 … P2n
… … … … …
n Pn1 Pn2 … Pnn

 ij
P
P

An m-step transition probability in a Markov chain is given
by
The m-step transition probability can be written as
)
|
(
)
(
i
X
j
X
P
P n
m
n
m
ij 

 
)
(m
ij
P
( ) ( )
1
, 0
n
m k m k
ij ir rj
r
P P P k m


   


Estimation of One-Step Transition Probabilities of
Markov Chain
• Transition probabilities of a Markov chain are estimated
using maximum likelihood estimate (MLE) from the
transition data (Anderson and Goodman, 1957).
• The MLE estimate of the transition probability Pij
(probability of moving from state i to state j in one step) is
given by
where Nij is number of cases in which Xn = i (state at time n
is i) and Xn+1 = j (state at time n + 1 is j).
1
ij
ij m
ik
k
N
P
N





Hypothesis Tests for Markov Chain:
Anderson Goodman Test
The null and alternative hypotheses to check whether the
sequence of random variables follows a Markov chain is
stated below
H0: The sequences of transitions (X1, X2, …, Xn) are
independent (zero-order Markov chain)
HA: The sequences of transitions (X1, X2, …, Xn) are
dependent (first-order Markov chain)
The corresponding test statistic is
 
  






 

 
m
i
n
j ij
ij
ij
E
E
O
1 1
2
2


Testing Time Homogeneity of Transition
Matrices: Likelihood Ratio Test
The test statistic is a likelihood test ratio statistic and is
given by (Anderson and Goodman, 1957):
( )
, ( )
ij
n t
ij
t i j
ij
P
P t


 
 
 
 
 
 


Stationary Distribution in a Markov Chain
Brand 1 Brand 2
Brand 1 0.80 0.20
Brand 2 0.25 0.75
Transition probability

Regular Matrix
A matrix P is called a regular matrix, when for some n, all
entries of Pn will be greater than zero, that is for some n
0

n
ij
P

Classification of States in a Markov
Chain

Accessible State
A state j is accessible (or reachable) from state i if there
exists a n such that that is, there exists a path
from state i to state j.
0

n
ij
P
Two states i and j are communicating states when there
exits n and m such that and That is, state j can
be reached (accessible) from state i and similarly state i can
be reached from state j. A Markov chain is called
irreducible if all states of the chain communicate with each
Communicating States

A state i of a Markov chain is called a recurrent state when
That is, if the state i is recurrent then the Markov chain will
visit state i infinite number of times. If state i is recurrent
and states i and j are communicating states, then state j is
also a recurrent state.
Recurrent and Transient States




1
n
n
ii
P

First passage time is the probability that the Markov chain
will enter state i exactly after n steps for the first time after
leaving state i, that is
Mean recurrence time is the average time taken to return to
state i after leaving state i. Mean recurrence time ii is
given by
First Passage Time and Mean Recurrence
Time
 
0
, , 1,2,..., 1|
n
ii n k
f P X i X i k n X i
     
 


1
n
n
ii
ii f
n

If the mean recurrence time is finite (ii is finite), then the
recurrent state is called a positive recurrent state and if it
is infinite then it is called null-recurrent state.

Periodic state is a special case of recurrent state in
which d(i) is the greatest common divisor of n such
that
If d(i) = 1, it is called aperiodic state and if d(i) 
2, then it is called a periodic state.
Periodic State
0

n
ii
P

Ergodic Markov Chain
A state i of a Markov chain is ergodic when it is positive
recurrent and aperiodic. Markov chain in which all states
are positive recurrent and aperiodic is called an ergodic
Markov chain.
For an ergodic Markov chain, a stationary distribution exists
that satisfies the system of equations
kj
m
k
k
j P


1

 1
1



m
k
k


In a Markov chain, the limiting probability is given by:
The main difference between limiting probability and
stationary distribution is that, stationary distribution when
exists is unique. Whereas limiting probability may not be
unique.
Limiting Probability
n
ij
n
p
Lt



A state i of a Markov chain is called an absorbing
state when Pii = 1, that is if the system enters this
state, it will remain in the same state.
Markov Chains with Absorbing States

Canonical Form of the Transition Matrix
of an Absorbing State Markov Chain
P =
A T
A I 0
T R Q

The matrix P is divided into 4 matrices I, 0, R, and Q, where
 Matrix I is the identity matrix. It corresponds to transition
within absorbing states.
 Matrix 0 is a matrix in which all elements are zero. Here
the elements correspond to transition between an
absorbing state and transient states.
 Matrix R represents the probability of absorption from a
transient state to an absorbing state.
 Matrix Q represents the transition between transient
states.

The expected duration to reach a state j from state i can be
calculated by solving the following system of equations:
Then, Ei,j satisfies the following system of equations:
Expected Duration to Reach a State from
Other States
, , ,
1 ,
i j i k k j
k
E P E i i j
   


Calculation of Retention Probability and
Customer Lifetime Value using Markov
Chains
CLV is the net present value (NPV) of the future margin
generated from its customers or customer segments. CLV
is calculated usually at a customer segment level.
Ching et al. (2004) showed that the steady-state retention
probability can be calculated using
where Rt is the steady-state retention probability.
  0 00
0
1 0
1
(1 )
1 1
1
n
i
t i
n
i
j
j
P
R P


  
   
 
 

 
 



The customer lifetime value for N periods is given by
(Pfeifer and Carraway, 2000):
where PI is the initial distribution of customers in different
states, P is the transition probability matrix, R is the
reward vector (margin generated in each customer
segments).
0
CLV
(1 )
N
t
t i




t
I
P × P R

Example
Calculate the steady-state retention probability and CLV for
6 periods (N = 5) using a discount factor of d = 0.95.
0 1 2 3 4
0 0.80 0.10 0.10 0 0
1 0.10 0.60 0.20 0.10 0
2 0.15 0.05 0.75 0.05 0
3 0.20 0 0.10 0.60 0.10
4 0.30 0 0 0.05 0.65

Revenue generated in different states
State 0 1 2 3 4
Average Margin 0 120 300 450 620
Customers
(in millions)
55.8 6.5 4.1 2.3 1.6

Solution
The stationary distribution equations are
• 0 = 0.80 + 0.101 + 0.152 + 0.203 + 0.304
• 1 = 0.10 + 0.601 + 0.052
• 2 = 0.10 + 0.21 + 0.752 + 0.103
• 3 = 0.101 + 0.052 + 0.603 + 0.054
• 0 + 1 + 2 + 3 + 4 = 1
Solving the above system of equations, we get 0 = 0.4287.
The steady-state retention probability Rt is
0 00
0
(1 ) 0.4287 (1 0.80)
1 1 0.85
1 1 0.4287
t
P
R
   
    
  

Customer lifetime value for N = 5 is
where
PI =
Substituting the values in CLV equation, we get CLV =
40181.59.

 


5
0 )
1
(
CLV
t
t
i
R
P
P t
I
 
6
.
1
3
.
2
1
.
4
5
.
6
8
.
55

















620
450
300
120
0
R

Markov Decision Process (MDP)
• Markov decision process (MDP) is a powerful technique
based on Markov chain that can be used for analyzing
any sequential decision-making scenario over a planning
horizon.
• MDP is defined using a 5-tuple (S, A, Psa, R, ) where
• S is the state space where S = {S1, S2, …, Sn}
• A is the set of actions A = {a1, a2, …, am}. Not all
actions will be available for all states. For example, if
MDP is used for buying and selling shares, the action
‘sell’ will be available to the decision maker only when
he/she owns a stock.

• Psa are state transition probability values that depend
on the current state and the chosen action.
• R is the reward vector that is a mapping of S × A 
R. The reward depends on the state and the action
chosen.
•  is the discount factor,   [0, 1]. Discount factor is
used since the rewards are received at future times.
where i is the interest rate.
i


1
1


Illustration of Markov decision process

Policy Iteration Algorithm
• A policy  in a MDP is a function that maps states to
actions, that is, : S  A.
• The objective of the policy iteration algorithm is to find an
optimal policy that will maximize the value function of the
Markov decision process (NPV of the future reward).
• The value function for policy  stating at state Si is given
by


 


 




reward
future
Discounted
reward
Immediate
)
(
)
(
)
( 




 


S
j
ij j
V
P
i
R
i
V 

• The policy iteration algorithm has two steps (Howard,
1971): (a) Policy evaluation and (b) policy improvement.
• Policy Evaluation Step:
Let S = {S1, S2, …, Si, …, Sn} be the set of states and A
= {a1, a2, …, am} be the set of actions. Write the system
of the value function V(i) equations for states {S1, S2,
…, Si, …, Sm}
1
( ) ( ) ( )
n
S
ij
i S
V i R i P V j i
   

    
 S

• Policy Improvement Step: In the policy improvement
step, the objective is to improve the value function obtained
by solving the system of equations in Eq.by finding a better
policy if it exists. In the policy improvement, the following
steps are used:
• Step 1: For i  S, compute
where
ai,new = A new action chosen for state i. = Value
function when the current policy for state i is changed to
ai,new. R(i, ai,new) = Reward when action for state i is
replaced with new action ai,new.









 



n
i
S
S
j
i
i
a
j
V
a
i
j
P
a
i
i
T
1
new
,
new
)
(
)
,
|
(
)
,
(
Max
)
( new
,
new
, 
R
)
(
new
i
T 

• Step 2 :If > V (i), then replace the policy 
with the new policy new (that is, action ai in policy
 is replaced with ai,new) else retain policy .
Repeat step 1 and 2 till all states and actions are
exhausted for a state.
)
(
new
i
T

Value Iteration Algorithm
Value iteration algorithm is basically a dynamic
programming algorithm which uses divide-and-conquer
strategy to solve the problem.
Dynamic programming is used when the problem fits into
the following characteristics:
 Problem can be divided into stages. In MDP, the
stages are different time periods during which decisions
need to be taken.
 The system is identified in each stage using a state.
In MDP, state is the information required to take
decision and find the optimal solution up to that stage.
 The decision taken at each stage updates the state at
the next stage of the planning horizon.

Bellman’s principle of optimality allows the problem to be
broken into smaller problems which can be solved easily.
The following steps are used in value iteration algorithm:
1) Identify the stages of the MDP (time periods).
2) Identify the state at the beginning of each stage
(information needed for choosing action).
3) Identify the actions (decisions) available for the decision
maker at each state.
4) Write the recursive relationship between the optimal
action (decision) at current stage and the previous
optimal action.
5) Optimal action set is the optimal policy for the problem
up to that stage.
Value Iteration Algorithm – Steps

The dynamic programming recursive equation
for the value iteration algorithm is given by
* *
1
1
( ) Max ( , ) ( ) ( )
i
n
t i ij i t
a
j
V i i a P a V j



 
  
 
 

A
R
Value Iteration Algorithm – Equation

Summary
 Most problems in analytics are dynamic in nature and
thus require collection of random variables to model the
problem.
 Stochastic process is a collection of random variables
usually indexed by time t and used while modelling
problems that are not independent and identically
distributed.
 Possion process is a counting process that is used in
decision making scenarios such as capacity planning and
spare parts demand forecasting. Compound Poisson
process can be used to study problems such as cash
replenishments at ATMs and total insurance claims etc,

 Markov chain is one of the most powerful models in
analytics with applications across industry sectors.
Google’s PageRank algorithm is based on Markov chain.
 Asset availability, market share, customer retention
probability and customer lifetime value are few
applications of Markov chain in analytics.
 Absorbing state Markov chain can be used for modelling
problem such as non-performing assets and customer
churn.
 Markov decision process in a reinforcement learning
algorithm with applications in sequential decision making
context. MDP is one of the best tools for making
sequential decision making under uncertainty

References
• Anderson T W and Goodman L A (1957), “Statistical
Inference about Markov Chain”, The Annals of
Mathematical Statistics, 28(1), 89110.
• Brin S and Page L (1998), “The Anatomy of a large-scale
hypertextual web search engine”, Computer Networks and
ISDN Systems, 30, 107117.
• Bellman R (1957), “Dynamic Programming”, Princeton
University Press, Princeton.
• Bayes B (2013), “First Links in the Markov Chain”,
American Scientist, 101, 9197.
• Cinlar E (1975). “Introduction to Stochastic Processes”,
Dover Publications, New York.

• Haggstrom O (2007), “Problem Solving is Often a Matter of
Cooking up an Appropriate Markov Chain”, Chalmers University
Report, available at
http://math.uchicago.edu/~shmuel/Network-course-
readings/Markov_chain_tricks.pdf, accessed on 15 May 2017.
• Howard R A (1971), “Dynamic Probabilistic Sysyems Volme II:
Semi-Markoc and Decision Processes”, John Wiley and Sons,
New York.
• Howard R (2002), “Comment on Origin and Application of
Markov Decision Processes”,Operations Research<AQ>Please
provide the journal’s name</AQ> 50(1), 100102.
• Ross S M (2010), “Introduction to Probability Models”, 10th
Edition, Academic Press, USA
• Styan G P H and Smith H (1964), “Markov Chain Applied to
Marketing”, Journal of Marketing Research, 1(1), 5055.

Business Analytics CHAPTER 16 STOCHASTIC MODELS Business Analytics.PPTX

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Business Analytics CHAPTER 16 STOCHASTIC MODELS Business Analytics.PPTX

Similar to Business Analytics CHAPTER 16 STOCHASTIC MODELS Business Analytics.PPTX (20)

Recently uploaded

Recently uploaded (20)

Business Analytics CHAPTER 16 STOCHASTIC MODELS Business Analytics.PPTX