SlideShare a Scribd company logo
Examination Number: B025068
1
Matriculation Number: s1121396
Examination Number: B025068
Decision-Making under Uncertainty
BUST10013
Course Organiser: Professor Tom Archibald
Word Count: 2100
Date of Submission: 27/11/2014
TURNCO EENGINEERING PROJECT
Examination Number: B025068
2
Table of Contents
Introduction
Part A:
1. Assumption, Solution and Analysis
1.1 Assumptions
1.2 Problem Formulation ( in Appendix i)
2. Model Solution and Solution Analysis
3.1 Model Solution for Scenario A & B & C
3.2 Solution Sensitivity Analysis
3.2.1 Robustness of repair length
3.2.2 Robustness to repair cost and preventive maintenance
3.2.3 Robustness to discount factor
Part B: Discussion- Strength and Weaknesses of the Methodology
Part C: Formulate a New Model with Replacement Equipment
Reference
Appendices:
Examination Number: B025068
3
Appendix i: Problem Formulation and Mathematical Formula
Appendix ii: Value Iteration Results for Scenario A, Scenario B and Scenario C
Appendix iii: Proportion of Time “operate” and “preventive maintenance”
Appendix iv: Sensitivity Analysis (Part A)
Appendix v: Replacement Results and Sensitivity Analysis (Part C)
Examination Number: B025068
4
Turnco Engineering Project
1. Introduction
With the rise in awareness of equipment management, knowledge of how to operate
equipment that is subject to deterioration and failure so as to maximize the rewards for
different conditions becomes more crucial. The Turnco Engineering machine shop faces this
problem with its grinding equipment. The aim of this report is to model the Turnco
Engineering equipment operation as a Markov decision process and try to use the Value
Iteration method in Excel to find the unique optimal policy. This report compares three
value iterations that is scenarios A, B, and C, with different revenue patterns. We obtain the
optimal policy for each revenue pattern A, B, C and test the robustness of our solution to
the following factors: (1) Repair length for A & B & C for day 1,2,3,4. ; (2) Repair cost for A &
B & C and Preventive maintenance cost for A & B & C; (3) Discount factor. It is found in the
Excel analysis that our model is sensitive to repair length and costs but robust to the change
in discount factor. In Part A, the assumption of the model will be clearly described; Problem
Formulation for Part A is in Appendix i . Most importantly, we provide the solution of our
model and the corresponding Sensitivity Analysis. In Part B, the strengths and weaknesses of
Examination Number: B025068
5
our model will be analysed in terms of the validity of the assumption, data and decision
criterion used in this model. In Part C, we will reformulate the model for new decision:
replacement of equipment.
Part A: Assumption, Solution and Analysis
2. Assumptions and Problem Formulation
There are 2 sections: section 2.1 is the assumptions of the model; section 2.2 is Problem
Formulation and Mathematical Formula. All section 2.2 is in Appendix i.
2.1 Assumptions:
 Assumption of Transition Probability: it is assumed that the transition probability
provided by the manufacturer is accurate and reliable, which is gaining by employing
data on the use of this equipment by customers over many years.
 Assumption on the Preventive Maintenance: we made an assumption that the
preventive maintenance of condition 2 will definitely lead to a 50% chance for
condition 1 and a 50% chance for condition 2. In addition, it is assumed that
preventive maintenance will be complete by the end of the day. So the costs for
each time preventive maintenance are the same as £300.
 Assumption on the Manufacturer Repair: We assumed that the only decision we can
make for condition 𝑖 = −3, −2, −1, 0 is manufacturing repair (See Appendix I, i= -
3…0 represent the repair conditions). At the same time, we assumed that failures
Examination Number: B025068
6
are instantaneously detected and the maximum time for manufacturer repair is
limited to 4 days. In other words, the consultant’s decision is allocated the same
maximum time-span as that of the workshop manager. Hence, for the sensitivity
analysis, we assumed that the consultant’s probability of the length of the
manufacturer repair is
 a
s
f
ollows (Table A.1):
Table A.1
 Assumption of Discounting: it is assumed that future rewards are discounted
according to a discount factor λ, with λ= 0.95.
3. Model Solution and Solution Analysis
We present here two sections: 3.1, the solution of our model from Excel; and 3.2, the solution
analysis.
3.1 Model Solution for Scenario A , B, C
By modelling the grinding equipment’s problems using the value iteration process, we
obtained optimal results for the different scenarios A, B, and C.
Length of manufacturer
repair (days)
1 2 3 4
Probability 0.5 0.25 0.125 0.0625
Examination Number: B025068
7
For scenarios A and B, the optimal policy is: operate the equipment (i.e.,”kp” in Excel) for all
working conditions from 1 to 5, and repair the equipment once it fails down. The optimal
policy for scenario C, however, is different from A and B. It shows that preventive
maintenance (i.e.,”pm” in Excel) is the optimal policy for working conditions 2 and 3, while
for other working conditions, “operate” is still the optimal policy. At the same time, the
error term between our solutions and optimal policy declines to 0.000000 when the number
of iterations stops at 477, which means that for larger n equals to 477, the infinite horizon
discount rewards approaches a stationary optimal policy; the value of each condition is
shown in Table A.2. For each scenario, the expected discounted rewards peak at condition 1,
at the time when each machine is new with the lowest failure probability. As for the
proportion of time to be operate or out of use, it is listed in Table A.4. The method used
here is global balance equations and the detailed description is shown in Appendix ii.
1
Decision of “Operate” is denoted as “kp” in our Excel;
“Repair” is denoted as “re” in Excel;
“Preventive Maintenance” is denoted as “pm” in Excel
“Replacement” is denoted as “rep” in Excel (Only for Part C)
CONDITIO
N
-3 -2 -1 0 1 2 3 4 5
OPTIMAL
POLICY OF
A
repai
r
repai
r
repai
r
repai
r
Operat
e 1
Operate Operate Operat
e
Operat
e
OPTIMAL
POLICY OF
B
repai
r
repai
r
repai
r
repai
r
Operat
e
Operate Operate Operat
e
Operat
e
OPTIMAL
POLICY OF
C
repai
r
repai
r
repai
r
repai
r
Operat
e
Preventive
Maintenanc
e
Preventive
Maintenanc
e
Operat
e
Operat
e
Optimal Policy Proportion of Time
to “Operate”
Proportion of Time
to be “our of use”
Examination Number: B025068
8
Table A.4
Table A.2
3.2 Solution Sensitivity Analysis
We conducted sensitivity analysis to test the robustness of our solution to the following factors: (1)
Repair length for A & B & C for day 1,2,3,4. ; (2) Repair cost for A & B & C and Preventive
maintenance cost for A & B & C; (3) Discount factor.
Scenario A 73.02% 26.98%
Scenario B 73.02% 26.98%
Scenario C 87.99% 12.01%
Examination Number: B025068
9
3.2.1 Robustness to the repair length
Based on the previous assumption that there is a limit of only four days to complete the
repair, we established a comparison between consultant and manager, in order to test
whether our solution is sensitive to the probability of the repair length (3,2,1,0). To be more
specific, there are 4 pairs of comparison groups. For 1 day to repair, the solution is robust to
change for scenarios A and B but in working condition 3 for scenario C, the optimal decision
is to carry out “operate” instead of “preventive maintenance”. If the repair length is 2 days,
the estimated probability from consultant and manager are the same, so there is no
sensitivity analysis of 2 days. However, If the repair length is 3 days, the optimal policy
changes from operate to preventive maintenance. From the Table A.5, we can conclude that
this model is sensitive to the changes of the repair length, especially for the 3 days’ repair.
(Note: Probability of repair in 2 days is same for consultant and manager, so there is no
sensitivity analysis for it.)
Length of
manufacturer repair
(days)
1 2 3 4
Consultant 0.5 0.25
(Same as Manager)
0.125 0.0625
Manager 0.45 0.25 0.20 0.10
Examination Number: B025068
10
Table A.3
Examination Number: B025068
11
Table A.5
3.2.2 Robustness to repair cost and preventive replacement cost
We did a sensitivity analysis for the repair costs (£500) and preventive replacement cost
(£300) respectively. As for repair costs (Table A.6), scenarios A and B are sensitive to costs
Examination Number: B025068
12
higher than £500; by contrast, scenario C is more sensitive to costs lower than £500. In
other words, when repair costs become higher and higher, preventive maintenance is a
better option compared with keeping “operate”. To be more specific, preventive
maintenance will improve the condition, decreasing the failure probability, and in turn
decreasing the possibility of needing to pay for repair costs. However, scenario C already has
two instances of preventive maintenance in working conditions 2 and 3, so the positive
influence to C is when the repair cost is lower, changing the optimal policy for condition 3
from “preventive maintenance” to “operate” at the expense of taking the risk of repair
(hence, repair cost); taking this risk is valuable as the repair cost has a large probability of
being lower than preventive cost. As for preventive costs (Table A.7), optimal policy for A
and B in condition 3 changes from “operate (kp)” to “preventive maintenance (pm)” when
the preventive costs are £100 or £150. Indeed, it is reasonable in reality to overcome failure
probability by adding preventive maintenance when its costs are relatively low. Also, it is
rational to decide on “Operate (kp)” rather than “preventive Maintenance “pm” when the
preventive costs are relatively high, i.e., if the costs are higher than £350, it is then not
worth doing preventive maintenance.
Table A.6
Examination Number: B025068
13
Table A.7
Examination Number: B025068
14
3.2.3 Robustness to discount factor
It is assumed that discount factor is rational for value larger than 0.95. So the range of
discount factor in sensitivity analysis is [0.95, 1] with increment equals to 0.005. (Table A.8)
According to the sensitivity test result for scenario A & B & C, we obtained that discount
factor is robust to our model. However, when the discount factor is larger than 0.98, the
error term is larger than 1, which means the model’s solutions are not good approximations
of the optimal policies.
Table A.8
Examination Number: B025068
15
Part B: Discussion – Strengths and Weaknesses of the Methodology
The results of the sensitivity analysis of repair length in Part A demonstrated a weakness in
the model: it is sensitive to small changes in probability, but, in reality, it is impractical to be
truly certain about one probability, especially within the difference of 0.075 (difference
between 0.2 and 0.125 given by the workshop manager and consultant respectively). This
presents a challenge when the data is not reliable; hence, the results are subject to
fluctuations. See Appendix iv for results of the sensitivity analysis.
At the same time, there is a limitation of this model: the overloading of equipment in reality.
Because there is no information provided from Turnco Engineering about the maximum
working capacity (hours) for the equipment, so this circumstance is out of consideration in
our model. However, if one piece of equipment with revenue pattern A follows our optimal
policy: always keeping “operate” for the whole range of conditions, it might lead to overload
working. Specifically, the failure probability might be higher than Fi, and it is then highly
likely that the equipment will fail down suddenly by accident. In order to avoid this
circumstance happens, we recommend this company to provide us with maximum working
capacity.
The strength of this model is that it considers all possible trends for the revenue from
operations: steady decline, steep initial decline and slow initial decline, so that it decreases
the effect of uncertain revenues on our optimal policy. Hence, it is provides a convenient
and precise means of implementing the results for decision-makers. In addition, our model
provides us with the optimal policy when the costs are uncertain (£100-£1000). Another
strength of the model is that the assumptions we made were both rational to the model and
Examination Number: B025068
16
practical to the real-world situation; this led to a direct implementation of the model
without limitation in reality.
However, there are four models which provide the best understanding of infinite-horizon
Markov decision problems: (1) Value Iteration; (2) Policy Iteration; (3) Modified Policy
Iteration; (4) Linear Programming. So whether value iteration performs as the best
“expected total discounted reward optimality criterion” is subject to consideration.
Puternman (1994) argued that value iteration should never be used. However, policy
iteration requires little additional programming yet attains superior convergence to the
optimal solution.
Part C: Formulate a New Model with Replacement Equipment
We established a new parameter “rep” represent for replacement cost. It is assumed that
the initial immediate rewards of replacement are £2000: a combination of replacement cost,
delivery cost and scrap value. Then we use the sensitivity analysis to test the suitable costs
that would be worth for a replacement. At the same time, the replacement order needed a
delivery time of 3 days until it was received, so we added the new transition probability.
(Table C.1). Thus, the whole transition probability is in Table C.3)
Table C.1
𝑝 𝑝,𝑝 -3 -2 -1 0 1 2 3 4 5
-2 0 0 1 0 0 0 0 0 0
-1 0 0 0 1 0 0 0 0 0
Examination Number: B025068
17
0 0 0 0 0 1 0 0 0 0
Table C.3
The new optimal decision for scenarios A , B, and C shows the same results as Part A.
However, after doing a sensitivity analysis for the replacement costs, the optimal policy
changes. We found that when the replacement cost is below £1000, the optimal policy is to
replace the equipment in condition 2. (Table C.2) The result is shown in Appendix iv.
Examination Number: B025068
18
Table C.2
Reference:
Martin L. Puterman. (1994). "Markov Decision Processes: Discrete Stochastic Dynamic
Programming." In: John Wiley & Sons, Inc. ISBN: 0471619779.
Appendix i: Problem Formulation and Mathematical Formula
Problem Formulation:
The Turnco Engineering shop operates grinding equipment and each item of equipment is
inspected by the managers at the end of the day. They must decide whether to continue to
operate the equipment, or whether preventive maintenance (pm) is needed. If the
equipment fails it has to be repaired by the manufacturer, so we assumed that failures are
instantaneously detected. Each item of equipment has five working conditions i = 1,2,3,4,5,
where 5 is the worst working condition. But according to the failure probability FI , when
the working condition moves from i to below the standard required normal condition, it has
4 possible repair lengths: 1 day, 2days, 3days, 4days, matching with the state S = {0,-1,-2,-3}
respectively. If the equipment is in preventive maintenance, it means that the equipment’s
condition will improve from state i to i − 1, except for working conditions 1 and 2.
Specifically, In working condition 1, there is no effect on preventive maintenance. In
Examination Number: B025068
19
condition 2, there is only 50% chance that pm will succeed, otherwise, it remains in
condition 2. In addition, the cost of preventive maintenance is £300 per day and the repair
cost is £500 per day; the overall cost of repair is the number of days multiplied by £500 per
day. As it is assumed that preventive maintenance will be completed by the end of the day,
and so its cost is only £300 each time. Parameter formulation is in Table D.1
Examination Number: B025068
20
Table D.1
Examination Number: B025068
21
Mathematical Formula:
State: status of equipment at the end of the day. Either i satisfying 1 ≤ i ≤ 5 indicating
working condition i , or {-3,-2,-1,0} indicating the numbers of days remaining in
manufacturer repair. S = {−3, −2, −1,0,1,2,3,4,5}.
Decision: Whether to operate (kp), preventive maintenance (pm) or carry out manufacturer
repair (re). Ki = {kp, pm} for 1 ≤ i ≤ 5 and Ki = {re} for −3 ≤ i ≤ 0.
Immediate reward:
rikp, A = ZAi, rikp, B = ZBi, rikp, C = ZCi, 1 ≤ i ≤ 5
ripm, A = ripm, B = ripm, C = −300 1 ≤ i ≤ 5
rire, A = rire, B = rire, C = −500 ∗ i − 3 ≤ i ≤ 0
Transition Probabilities:
We have pi,j = qi,j for 1 ≤ i ≤ 5, 1 ≤ j ≤ 5, j ≥ i , which is the probability transition from
condition i to j. At the same time, the transition probability pi,j for 1 ≤ i ≤ 5, −3 ≤ j ≤ 0,
is the multiplication of Fi *prl_i , where Fi is the failure probability from any state i
(1 ≤ i ≤ 5) to below standard condition -3,-2,-1, 0. prl_k is the probability of the repair
length (rl): k= 1,2,3,4. In matrix notion, we have:
Fi = (
0.01
0.02
0.08
0.15
0.3
); prl_i = (0.45 0.25 0.20 0.10);
Hence, the transition probability Fi*prl_i gives us the probability of repair length for each
equipment’s working conditions.
Examination Number: B025068
22
pi,j -3 -2 -1 0
1 0.01 0.02 0.025 0.0045
2 0.002 0.004 0.005 0.009
3 0.008 0.016 0.02 0.036
4 0.0015 0.03 0.0375 0.068
5 0.03 0.06 0.075 0.135
In addition, under the preventive maintenance, the working condition improves from i to
the nearest upper condition i-1, except for conditions 1 and 2, that is pi,i−1 = 1 for i =
3,4,5; p1,1 = 1; p2,1 = 0.5. Furthermore, the condition after the repair is 1 for all.
Optimal Equation:
The main problem that we consider in this project is to find Vi, the maximum expected
discounted reward over an infinite horizon when the process is in state i initially. The
optimality equation is:
𝑣𝑖
𝑛
=
𝑚𝑎𝑥
𝑘 ∊ 𝐾𝑖
{𝑟𝑖
𝑘
+ 𝛽(𝑝𝑖,1
𝑘
𝑣1
𝑛−1
+ 𝑝𝑖,2
𝑘
𝑣2
𝑛−1
+ ⋯ + 𝑝𝑖,𝑀
𝑘
𝑣 𝑀
𝑛−1
)}
Examination Number: B025068
23
For Example: Formula used for v_n,-3,re is =r__3re_A+df*SUMPRODUCT(p__3re,B3:J3)
AH4=ABS(B4-B3)
F4=MAX(X4:Y4)
Examination Number: B025068
24
Immediate reward and transition probability for Part A and Part B only (Without
replacement)
Appendix ii: Value Iteration Results for Scenario A & B & C
(Beginning of the calculation and a few at the end of the calculation)
Scenario A:
Examination Number: B025068
25
Examination Number: B025068
26
Examination Number: B025068
27
Examination Number: B025068
28
Scenario B:
Examination Number: B025068
29
Examination Number: B025068
30
Scenario C:
Examination Number: B025068
31
Examination Number: B025068
32
Examination Number: B025068
33
Appendix iii: Proportion of Time “operate” and “preventive maintenance”
For Scenario A & B : The result is
Thus, according to the decision of the model, we can list the transition probability for
A&B with the optimal decision
𝒑𝒊,𝒋 -3 -2 -1 0 1 2 3 4 5
-3 0 1 0 0 0 0 0 0 0
CONDITION -3 -2 -1 0 1 2 3 4 5
OPTIMAL
POLICY OF
A
repair repair repair repair Operate Operate Operate Operate Operate
OPTIMAL
POLICY OF
B
repair repair repair repair Operate Operate Operate Operate Operate
Examination Number: B025068
34
-2 0 0 1 0 0 0 0 0 0
-1 0 0 0 1 0 0 0 0 0
0 0 0 0 0 1 0 0 0 0
1 0.001 0.002 0.0025 0.0045 0.95 0.03 0.01 0 0
2 0.002 0.004 0.005 0.009 0 0.9 0.05 0.02 0.01
3 0.008 0.016 0.02 0.036 0 0 0.8 0.08 0.04
4 0.015 0.03 0.0375 0.0675 0 0 0 0.7 0.15
5 0.03 0.06 0.075 0.135 0 0 0 0 0.7
Then, the Global Balance Equation is
𝝅−𝟑 = 𝟎. 𝟎𝟎𝟏𝝅 𝟏 + 𝟎. 𝟎𝟎𝟐𝝅 𝟐 + 𝟎. 𝟎𝟎𝟖𝝅 𝟑 + 𝟎. 𝟎𝟏𝟓𝝅 𝟒 + 𝟎. 𝟎𝟑𝝅 𝟓
𝝅−𝟐 = 𝟎. 𝟎𝟎𝟐𝝅 𝟏 + 𝟎. 𝟎𝟎𝟒𝝅 𝟐 + 𝟎. 𝟎𝟎𝟏𝟔𝝅 𝟑 + 𝟎. 𝟎𝟑𝝅 𝟒 + 𝟎. 𝟎𝟔𝝅 𝟓 + 𝝅−𝟑
𝝅−𝟏 = 𝟎. 𝟎𝟎𝟐𝟓𝝅 𝟏 + 𝟎. 𝟎𝟎𝟓𝝅 𝟐 + 𝟎. 𝟎𝟎𝟐𝝅 𝟑 + 𝟎. 𝟎𝟑𝟕𝟓𝝅 𝟒 + 𝟎. 𝟎𝟕𝟓𝝅 𝟓 + 𝝅−𝟐
𝝅 𝟎 = 𝟎. 𝟎𝟎𝟒𝟓𝝅 𝟏 + 𝟎. 𝟎𝟗𝝅 𝟐 + 𝟎. 𝟎𝟎𝟑𝟔𝝅 𝟑 + 𝟎. 𝟎𝟔𝟕𝟓𝝅 𝟒 + 𝟎. 𝟏𝟑𝟓𝝅 𝟓 + 𝝅−𝟏
𝝅 𝟏 = 𝝅 𝟎 + 𝟎. 𝟗𝟓𝝅 𝟏
𝝅 𝟐 = 𝟎. 𝟎𝟑𝝅 𝟏 + 𝟎. 𝟗𝝅 𝟐
𝝅 𝟑 = 𝟎. 𝟎𝟏𝝅 𝟏 + 𝟎. 𝟎𝟓𝝅 𝟐 + 𝟎. 𝟖𝝅 𝟑
𝝅 𝟒 = 𝟎. 𝟎𝟐𝝅 𝟏 + 𝟎. 𝟎𝟖𝝅 𝟐 + 𝟎. 𝟕𝝅 𝟑
𝝅 𝟓 = 𝟎. 𝟎𝟎𝟏𝝅 𝟐 + 𝟎. 𝟎𝟒𝝅 𝟑 + 𝟎. 𝟏𝟓𝝅 𝟒 + 𝟎. 𝟕𝝅 𝟓
𝝅−𝟑+𝝅−𝟐 + 𝝅−𝟏 + 𝝅 𝟎 + 𝝅 𝟏 + 𝝅 𝟐 + 𝝅 𝟑 + 𝝅 𝟒 + 𝝅 𝟓 = 𝟏
Totally, the mathematical result for A & B :
𝝅−𝟏 = 𝟎. 𝟏𝟐𝟓𝟔𝟗𝟑; 𝝅−𝟐 = 𝟎. 𝟏𝟏𝟖𝟐𝟐𝟕; 𝝅−𝟑 = 𝟎. 𝟎𝟎𝟐𝟗𝟖𝟐; 𝝅 𝟎 = 𝟎. 𝟎𝟐𝟐𝟗
𝝅 𝟏 = 𝟎. 𝟒𝟓𝟕𝟗𝟗𝟔; 𝝅 𝟐 = 𝟎. 𝟏𝟑𝟕𝟑𝟗𝟗; 𝝅 𝟑 = 𝟎. 𝟎𝟒𝟓𝟖; 𝝅 𝟒 = 𝟎. 𝟎𝟓𝟐𝟐𝟏𝟐; 𝝅 𝟓 = 𝟎. 𝟎𝟑𝟔𝟕𝟗𝟐
The proportion of time to be “operate” is
𝝅 𝟏 + 𝝅 𝟐 + 𝝅 𝟑 + 𝝅 𝟒 + 𝝅 𝟓 = 𝟎. 𝟕𝟑𝟎𝟏𝟗𝟗=73.02%
Examination Number: B025068
35
The proportion of time to be “out of use” is
𝝅−𝟏 + 𝝅−𝟐 + 𝝅−𝟑 + 𝝅 𝟎 = 𝟎. 𝟐𝟔𝟗𝟖𝟎𝟐=26.98%
For Scenario C : The result is
Thus, according to the decision of the model, we can list the transition probability for C
with the optimal decision
𝒑𝒊,𝒋 -3 -2 -1 0 1 2 3 4 5
-3 0 1 0 0 0 0 0 0 0
-2 0 0 1 0 0 0 0 0 0
-1 0 0 0 1 0 0 0 0 0
0 0 0 0 0 1 0 0 0 0
1 0.001 0.002 0.0025 0.0045 0.95 0.03 0.01 0 0
2 0 0 0 0 0.5 0.5 0 0 0
3 0 0 0 0 0 1 0 0 0
4 0.015 0.03 0.0375 0.0675 0 0 0 0.7 0.15
5 0.03 0.06 0.075 0.135 0 0 0 0 0.7
Then, the Global Balance Equation is
𝝅−𝟑 = 𝟎. 𝟎𝟎𝟏𝝅 𝟏 + 𝟎. 𝟎𝟏𝟓𝝅 𝟒 + 𝟎. 𝟎𝟑𝝅 𝟓
𝝅−𝟐 = 𝟎. 𝟎𝟎𝟐𝝅 𝟏 + 𝟎. 𝟎𝟑𝝅 𝟒 + 𝟎. 𝟎𝟔𝝅 𝟓 + 𝝅−𝟑
𝝅−𝟏 = 𝟎. 𝟎𝟎𝟐𝟓𝝅 𝟏 + 𝟎. 𝟎𝟑𝟕𝟓𝝅 𝟒 + 𝟎. 𝟎𝟕𝟓𝝅 𝟓 + 𝝅−𝟐
𝝅 𝟎 = 𝟎. 𝟎𝟎𝟒𝟓𝝅 𝟏 + 𝟎. 𝟎𝟔𝟕𝟓𝝅 𝟒 + 𝟎. 𝟏𝟑𝟓𝝅 𝟓 + 𝝅−𝟏
𝝅 𝟏 = 𝝅 𝟎 + 𝟎. 𝟗𝟓𝝅 𝟏 + 𝟎. 𝟓𝝅 𝟐
CONDITIO
N
-3 -2 -1 0 1 2 3 4 5
OPTIMAL
POLICY OF
A
repai
r
repai
r
repai
r
repai
r
Operat
e
Preventive
Maintenanc
e
Preventive
Maintenanc
e
Operat
e
Operat
e
Examination Number: B025068
36
𝝅 𝟐 = 𝟎. 𝟎𝟑𝝅 𝟏 + 𝟎. 𝟓𝝅 𝟐 𝝅 𝟑
𝝅 𝟑 = 𝟎. 𝟎𝟏𝝅 𝟏
𝝅 𝟒 = 𝟎. 𝟕𝝅 𝟒
𝝅 𝟓 = 𝟎. 𝟏𝟓𝝅 𝟒 + 𝟎. 𝟕𝝅 𝟓
𝝅−𝟑+𝝅−𝟐 + 𝝅−𝟏 + 𝝅 𝟎 + 𝝅 𝟏 + 𝝅 𝟐 + 𝝅 𝟑 + 𝝅 𝟒 + 𝝅 𝟓 = 𝟏
Totally, the mathematical result for A & B :
𝝅−𝟏 = 𝟎. 𝟎𝟏𝟐𝟕𝟓𝟖𝟒𝟎𝟓; 𝝅−𝟐 = 𝟎. 𝟎𝟏𝟎𝟓𝟐𝟖𝟔𝟖; 𝝅−𝟑 = 𝟎. 𝟎𝟎𝟎𝟖𝟕𝟗𝟖𝟗; 𝝅 𝟎 = 𝟎. 𝟎𝟏𝟔𝟕𝟏𝟗𝟕𝟏
𝝅 𝟏 = 𝟎. 𝟖𝟕𝟗𝟖𝟗; 𝝅 𝟐 = 𝟎. 𝟎𝟕𝟎𝟑𝟗𝟏𝟓𝟓; 𝝅 𝟑 = 𝟎. 𝟎𝟎𝟖𝟕𝟗𝟖𝟗; 𝝅 𝟒 = 𝝅 𝟓 = 𝟎
The proportion of time to be “operate” is
𝝅 𝟏 + 𝝅 𝟒 + 𝝅 𝟓 =0.87989=87.99%
The proportion of time to be “out of use” is
𝝅−𝟏 + 𝝅−𝟐 + 𝝅−𝟑 + 𝝅 𝟎 + 𝝅 𝟐 + 𝝅 𝟑 =0.12011=12.01%
Appendix iv: Sensitivity Analysis
 Sensitivity Analysis of repair length of A&B&C for day 1,2,3,4.
Examination Number: B025068
37
 Sensitivity Analysis for repair costs for A,B,C between £100-£1000
Examination Number: B025068
38
 Sensitivity Analysis of Discount Factor of A
 Sensitivity Analysis of Preventive Costs:
Examination Number: B025068
39
Appendix V: Replacement Results and Sensitivity Analysis (Part C)
Examination Number: B025068
40

More Related Content

Similar to DMU_Yanran_Zhu_s1121396.docx

SJSoM Pratiman Case Study Competition 2013
SJSoM Pratiman Case Study Competition 2013SJSoM Pratiman Case Study Competition 2013
SJSoM Pratiman Case Study Competition 2013
Rajib Layek
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
Young Alista
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
Luis Goldster
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
Fraboni Ec
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
Harry Potter
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
Tony Nguyen
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
James Wong
 
Production & Operation Management Chapter13[1]
Production & Operation Management Chapter13[1]Production & Operation Management Chapter13[1]
Production & Operation Management Chapter13[1]
Hariharan Ponnusamy
 
Chapter13[1]
Chapter13[1]Chapter13[1]
Chapter13[1]
Hariharan Ponnusamy
 
Om0015 maintenance management..
Om0015   maintenance management..Om0015   maintenance management..
Om0015 maintenance management..
smumbahelp
 
Om0015 maintenance management..
Om0015   maintenance management..Om0015   maintenance management..
Om0015 maintenance management..
smumbahelp
 
Electiva ii
Electiva iiElectiva ii
Electiva ii
Elyseth marin
 
Cpk guide 0211_tech1
Cpk guide 0211_tech1Cpk guide 0211_tech1
Cpk guide 0211_tech1
Piyush Bose
 
MA909 report_RiskGlobalX
MA909 report_RiskGlobalXMA909 report_RiskGlobalX
MA909 report_RiskGlobalX
Manling Zhang
 
Envse 470 final project
Envse 470 final projectEnvse 470 final project
Envse 470 final project
SarahDent10
 
Dynamic future flare_design2020
Dynamic future flare_design2020Dynamic future flare_design2020
Dynamic future flare_design2020
Armin Mn
 
Tuning the model predictive control of a crude distillation unit
Tuning the model predictive control of a crude distillation unitTuning the model predictive control of a crude distillation unit
Tuning the model predictive control of a crude distillation unit
ISA Interchange
 
Protcy inter eng-giurca_ppt_v4
Protcy inter eng-giurca_ppt_v4Protcy inter eng-giurca_ppt_v4
Protcy inter eng-giurca_ppt_v4
Ioan Giurca
 
test 1Suppose that you are using the simple mean to make a forecas.docx
test 1Suppose that you are using the simple mean to make a forecas.docxtest 1Suppose that you are using the simple mean to make a forecas.docx
test 1Suppose that you are using the simple mean to make a forecas.docx
todd191
 
Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3 Complex Designs
Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3  Complex Designs Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3  Complex Designs
Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3 Complex Designs
J. García - Verdugo
 

Similar to DMU_Yanran_Zhu_s1121396.docx (20)

SJSoM Pratiman Case Study Competition 2013
SJSoM Pratiman Case Study Competition 2013SJSoM Pratiman Case Study Competition 2013
SJSoM Pratiman Case Study Competition 2013
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
 
Decision analysis
Decision analysisDecision analysis
Decision analysis
 
Production & Operation Management Chapter13[1]
Production & Operation Management Chapter13[1]Production & Operation Management Chapter13[1]
Production & Operation Management Chapter13[1]
 
Chapter13[1]
Chapter13[1]Chapter13[1]
Chapter13[1]
 
Om0015 maintenance management..
Om0015   maintenance management..Om0015   maintenance management..
Om0015 maintenance management..
 
Om0015 maintenance management..
Om0015   maintenance management..Om0015   maintenance management..
Om0015 maintenance management..
 
Electiva ii
Electiva iiElectiva ii
Electiva ii
 
Cpk guide 0211_tech1
Cpk guide 0211_tech1Cpk guide 0211_tech1
Cpk guide 0211_tech1
 
MA909 report_RiskGlobalX
MA909 report_RiskGlobalXMA909 report_RiskGlobalX
MA909 report_RiskGlobalX
 
Envse 470 final project
Envse 470 final projectEnvse 470 final project
Envse 470 final project
 
Dynamic future flare_design2020
Dynamic future flare_design2020Dynamic future flare_design2020
Dynamic future flare_design2020
 
Tuning the model predictive control of a crude distillation unit
Tuning the model predictive control of a crude distillation unitTuning the model predictive control of a crude distillation unit
Tuning the model predictive control of a crude distillation unit
 
Protcy inter eng-giurca_ppt_v4
Protcy inter eng-giurca_ppt_v4Protcy inter eng-giurca_ppt_v4
Protcy inter eng-giurca_ppt_v4
 
test 1Suppose that you are using the simple mean to make a forecas.docx
test 1Suppose that you are using the simple mean to make a forecas.docxtest 1Suppose that you are using the simple mean to make a forecas.docx
test 1Suppose that you are using the simple mean to make a forecas.docx
 
Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3 Complex Designs
Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3  Complex Designs Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3  Complex Designs
Javier Garcia - Verdugo Sanchez - Six Sigma Training - W3 Complex Designs
 

DMU_Yanran_Zhu_s1121396.docx

  • 1. Examination Number: B025068 1 Matriculation Number: s1121396 Examination Number: B025068 Decision-Making under Uncertainty BUST10013 Course Organiser: Professor Tom Archibald Word Count: 2100 Date of Submission: 27/11/2014 TURNCO EENGINEERING PROJECT
  • 2. Examination Number: B025068 2 Table of Contents Introduction Part A: 1. Assumption, Solution and Analysis 1.1 Assumptions 1.2 Problem Formulation ( in Appendix i) 2. Model Solution and Solution Analysis 3.1 Model Solution for Scenario A & B & C 3.2 Solution Sensitivity Analysis 3.2.1 Robustness of repair length 3.2.2 Robustness to repair cost and preventive maintenance 3.2.3 Robustness to discount factor Part B: Discussion- Strength and Weaknesses of the Methodology Part C: Formulate a New Model with Replacement Equipment Reference Appendices:
  • 3. Examination Number: B025068 3 Appendix i: Problem Formulation and Mathematical Formula Appendix ii: Value Iteration Results for Scenario A, Scenario B and Scenario C Appendix iii: Proportion of Time “operate” and “preventive maintenance” Appendix iv: Sensitivity Analysis (Part A) Appendix v: Replacement Results and Sensitivity Analysis (Part C)
  • 4. Examination Number: B025068 4 Turnco Engineering Project 1. Introduction With the rise in awareness of equipment management, knowledge of how to operate equipment that is subject to deterioration and failure so as to maximize the rewards for different conditions becomes more crucial. The Turnco Engineering machine shop faces this problem with its grinding equipment. The aim of this report is to model the Turnco Engineering equipment operation as a Markov decision process and try to use the Value Iteration method in Excel to find the unique optimal policy. This report compares three value iterations that is scenarios A, B, and C, with different revenue patterns. We obtain the optimal policy for each revenue pattern A, B, C and test the robustness of our solution to the following factors: (1) Repair length for A & B & C for day 1,2,3,4. ; (2) Repair cost for A & B & C and Preventive maintenance cost for A & B & C; (3) Discount factor. It is found in the Excel analysis that our model is sensitive to repair length and costs but robust to the change in discount factor. In Part A, the assumption of the model will be clearly described; Problem Formulation for Part A is in Appendix i . Most importantly, we provide the solution of our model and the corresponding Sensitivity Analysis. In Part B, the strengths and weaknesses of
  • 5. Examination Number: B025068 5 our model will be analysed in terms of the validity of the assumption, data and decision criterion used in this model. In Part C, we will reformulate the model for new decision: replacement of equipment. Part A: Assumption, Solution and Analysis 2. Assumptions and Problem Formulation There are 2 sections: section 2.1 is the assumptions of the model; section 2.2 is Problem Formulation and Mathematical Formula. All section 2.2 is in Appendix i. 2.1 Assumptions:  Assumption of Transition Probability: it is assumed that the transition probability provided by the manufacturer is accurate and reliable, which is gaining by employing data on the use of this equipment by customers over many years.  Assumption on the Preventive Maintenance: we made an assumption that the preventive maintenance of condition 2 will definitely lead to a 50% chance for condition 1 and a 50% chance for condition 2. In addition, it is assumed that preventive maintenance will be complete by the end of the day. So the costs for each time preventive maintenance are the same as £300.  Assumption on the Manufacturer Repair: We assumed that the only decision we can make for condition 𝑖 = −3, −2, −1, 0 is manufacturing repair (See Appendix I, i= - 3…0 represent the repair conditions). At the same time, we assumed that failures
  • 6. Examination Number: B025068 6 are instantaneously detected and the maximum time for manufacturer repair is limited to 4 days. In other words, the consultant’s decision is allocated the same maximum time-span as that of the workshop manager. Hence, for the sensitivity analysis, we assumed that the consultant’s probability of the length of the manufacturer repair is  a s f ollows (Table A.1): Table A.1  Assumption of Discounting: it is assumed that future rewards are discounted according to a discount factor λ, with λ= 0.95. 3. Model Solution and Solution Analysis We present here two sections: 3.1, the solution of our model from Excel; and 3.2, the solution analysis. 3.1 Model Solution for Scenario A , B, C By modelling the grinding equipment’s problems using the value iteration process, we obtained optimal results for the different scenarios A, B, and C. Length of manufacturer repair (days) 1 2 3 4 Probability 0.5 0.25 0.125 0.0625
  • 7. Examination Number: B025068 7 For scenarios A and B, the optimal policy is: operate the equipment (i.e.,”kp” in Excel) for all working conditions from 1 to 5, and repair the equipment once it fails down. The optimal policy for scenario C, however, is different from A and B. It shows that preventive maintenance (i.e.,”pm” in Excel) is the optimal policy for working conditions 2 and 3, while for other working conditions, “operate” is still the optimal policy. At the same time, the error term between our solutions and optimal policy declines to 0.000000 when the number of iterations stops at 477, which means that for larger n equals to 477, the infinite horizon discount rewards approaches a stationary optimal policy; the value of each condition is shown in Table A.2. For each scenario, the expected discounted rewards peak at condition 1, at the time when each machine is new with the lowest failure probability. As for the proportion of time to be operate or out of use, it is listed in Table A.4. The method used here is global balance equations and the detailed description is shown in Appendix ii. 1 Decision of “Operate” is denoted as “kp” in our Excel; “Repair” is denoted as “re” in Excel; “Preventive Maintenance” is denoted as “pm” in Excel “Replacement” is denoted as “rep” in Excel (Only for Part C) CONDITIO N -3 -2 -1 0 1 2 3 4 5 OPTIMAL POLICY OF A repai r repai r repai r repai r Operat e 1 Operate Operate Operat e Operat e OPTIMAL POLICY OF B repai r repai r repai r repai r Operat e Operate Operate Operat e Operat e OPTIMAL POLICY OF C repai r repai r repai r repai r Operat e Preventive Maintenanc e Preventive Maintenanc e Operat e Operat e Optimal Policy Proportion of Time to “Operate” Proportion of Time to be “our of use”
  • 8. Examination Number: B025068 8 Table A.4 Table A.2 3.2 Solution Sensitivity Analysis We conducted sensitivity analysis to test the robustness of our solution to the following factors: (1) Repair length for A & B & C for day 1,2,3,4. ; (2) Repair cost for A & B & C and Preventive maintenance cost for A & B & C; (3) Discount factor. Scenario A 73.02% 26.98% Scenario B 73.02% 26.98% Scenario C 87.99% 12.01%
  • 9. Examination Number: B025068 9 3.2.1 Robustness to the repair length Based on the previous assumption that there is a limit of only four days to complete the repair, we established a comparison between consultant and manager, in order to test whether our solution is sensitive to the probability of the repair length (3,2,1,0). To be more specific, there are 4 pairs of comparison groups. For 1 day to repair, the solution is robust to change for scenarios A and B but in working condition 3 for scenario C, the optimal decision is to carry out “operate” instead of “preventive maintenance”. If the repair length is 2 days, the estimated probability from consultant and manager are the same, so there is no sensitivity analysis of 2 days. However, If the repair length is 3 days, the optimal policy changes from operate to preventive maintenance. From the Table A.5, we can conclude that this model is sensitive to the changes of the repair length, especially for the 3 days’ repair. (Note: Probability of repair in 2 days is same for consultant and manager, so there is no sensitivity analysis for it.) Length of manufacturer repair (days) 1 2 3 4 Consultant 0.5 0.25 (Same as Manager) 0.125 0.0625 Manager 0.45 0.25 0.20 0.10
  • 11. Examination Number: B025068 11 Table A.5 3.2.2 Robustness to repair cost and preventive replacement cost We did a sensitivity analysis for the repair costs (£500) and preventive replacement cost (£300) respectively. As for repair costs (Table A.6), scenarios A and B are sensitive to costs
  • 12. Examination Number: B025068 12 higher than £500; by contrast, scenario C is more sensitive to costs lower than £500. In other words, when repair costs become higher and higher, preventive maintenance is a better option compared with keeping “operate”. To be more specific, preventive maintenance will improve the condition, decreasing the failure probability, and in turn decreasing the possibility of needing to pay for repair costs. However, scenario C already has two instances of preventive maintenance in working conditions 2 and 3, so the positive influence to C is when the repair cost is lower, changing the optimal policy for condition 3 from “preventive maintenance” to “operate” at the expense of taking the risk of repair (hence, repair cost); taking this risk is valuable as the repair cost has a large probability of being lower than preventive cost. As for preventive costs (Table A.7), optimal policy for A and B in condition 3 changes from “operate (kp)” to “preventive maintenance (pm)” when the preventive costs are £100 or £150. Indeed, it is reasonable in reality to overcome failure probability by adding preventive maintenance when its costs are relatively low. Also, it is rational to decide on “Operate (kp)” rather than “preventive Maintenance “pm” when the preventive costs are relatively high, i.e., if the costs are higher than £350, it is then not worth doing preventive maintenance. Table A.6
  • 14. Examination Number: B025068 14 3.2.3 Robustness to discount factor It is assumed that discount factor is rational for value larger than 0.95. So the range of discount factor in sensitivity analysis is [0.95, 1] with increment equals to 0.005. (Table A.8) According to the sensitivity test result for scenario A & B & C, we obtained that discount factor is robust to our model. However, when the discount factor is larger than 0.98, the error term is larger than 1, which means the model’s solutions are not good approximations of the optimal policies. Table A.8
  • 15. Examination Number: B025068 15 Part B: Discussion – Strengths and Weaknesses of the Methodology The results of the sensitivity analysis of repair length in Part A demonstrated a weakness in the model: it is sensitive to small changes in probability, but, in reality, it is impractical to be truly certain about one probability, especially within the difference of 0.075 (difference between 0.2 and 0.125 given by the workshop manager and consultant respectively). This presents a challenge when the data is not reliable; hence, the results are subject to fluctuations. See Appendix iv for results of the sensitivity analysis. At the same time, there is a limitation of this model: the overloading of equipment in reality. Because there is no information provided from Turnco Engineering about the maximum working capacity (hours) for the equipment, so this circumstance is out of consideration in our model. However, if one piece of equipment with revenue pattern A follows our optimal policy: always keeping “operate” for the whole range of conditions, it might lead to overload working. Specifically, the failure probability might be higher than Fi, and it is then highly likely that the equipment will fail down suddenly by accident. In order to avoid this circumstance happens, we recommend this company to provide us with maximum working capacity. The strength of this model is that it considers all possible trends for the revenue from operations: steady decline, steep initial decline and slow initial decline, so that it decreases the effect of uncertain revenues on our optimal policy. Hence, it is provides a convenient and precise means of implementing the results for decision-makers. In addition, our model provides us with the optimal policy when the costs are uncertain (£100-£1000). Another strength of the model is that the assumptions we made were both rational to the model and
  • 16. Examination Number: B025068 16 practical to the real-world situation; this led to a direct implementation of the model without limitation in reality. However, there are four models which provide the best understanding of infinite-horizon Markov decision problems: (1) Value Iteration; (2) Policy Iteration; (3) Modified Policy Iteration; (4) Linear Programming. So whether value iteration performs as the best “expected total discounted reward optimality criterion” is subject to consideration. Puternman (1994) argued that value iteration should never be used. However, policy iteration requires little additional programming yet attains superior convergence to the optimal solution. Part C: Formulate a New Model with Replacement Equipment We established a new parameter “rep” represent for replacement cost. It is assumed that the initial immediate rewards of replacement are £2000: a combination of replacement cost, delivery cost and scrap value. Then we use the sensitivity analysis to test the suitable costs that would be worth for a replacement. At the same time, the replacement order needed a delivery time of 3 days until it was received, so we added the new transition probability. (Table C.1). Thus, the whole transition probability is in Table C.3) Table C.1 𝑝 𝑝,𝑝 -3 -2 -1 0 1 2 3 4 5 -2 0 0 1 0 0 0 0 0 0 -1 0 0 0 1 0 0 0 0 0
  • 17. Examination Number: B025068 17 0 0 0 0 0 1 0 0 0 0 Table C.3 The new optimal decision for scenarios A , B, and C shows the same results as Part A. However, after doing a sensitivity analysis for the replacement costs, the optimal policy changes. We found that when the replacement cost is below £1000, the optimal policy is to replace the equipment in condition 2. (Table C.2) The result is shown in Appendix iv.
  • 18. Examination Number: B025068 18 Table C.2 Reference: Martin L. Puterman. (1994). "Markov Decision Processes: Discrete Stochastic Dynamic Programming." In: John Wiley & Sons, Inc. ISBN: 0471619779. Appendix i: Problem Formulation and Mathematical Formula Problem Formulation: The Turnco Engineering shop operates grinding equipment and each item of equipment is inspected by the managers at the end of the day. They must decide whether to continue to operate the equipment, or whether preventive maintenance (pm) is needed. If the equipment fails it has to be repaired by the manufacturer, so we assumed that failures are instantaneously detected. Each item of equipment has five working conditions i = 1,2,3,4,5, where 5 is the worst working condition. But according to the failure probability FI , when the working condition moves from i to below the standard required normal condition, it has 4 possible repair lengths: 1 day, 2days, 3days, 4days, matching with the state S = {0,-1,-2,-3} respectively. If the equipment is in preventive maintenance, it means that the equipment’s condition will improve from state i to i − 1, except for working conditions 1 and 2. Specifically, In working condition 1, there is no effect on preventive maintenance. In
  • 19. Examination Number: B025068 19 condition 2, there is only 50% chance that pm will succeed, otherwise, it remains in condition 2. In addition, the cost of preventive maintenance is £300 per day and the repair cost is £500 per day; the overall cost of repair is the number of days multiplied by £500 per day. As it is assumed that preventive maintenance will be completed by the end of the day, and so its cost is only £300 each time. Parameter formulation is in Table D.1
  • 21. Examination Number: B025068 21 Mathematical Formula: State: status of equipment at the end of the day. Either i satisfying 1 ≤ i ≤ 5 indicating working condition i , or {-3,-2,-1,0} indicating the numbers of days remaining in manufacturer repair. S = {−3, −2, −1,0,1,2,3,4,5}. Decision: Whether to operate (kp), preventive maintenance (pm) or carry out manufacturer repair (re). Ki = {kp, pm} for 1 ≤ i ≤ 5 and Ki = {re} for −3 ≤ i ≤ 0. Immediate reward: rikp, A = ZAi, rikp, B = ZBi, rikp, C = ZCi, 1 ≤ i ≤ 5 ripm, A = ripm, B = ripm, C = −300 1 ≤ i ≤ 5 rire, A = rire, B = rire, C = −500 ∗ i − 3 ≤ i ≤ 0 Transition Probabilities: We have pi,j = qi,j for 1 ≤ i ≤ 5, 1 ≤ j ≤ 5, j ≥ i , which is the probability transition from condition i to j. At the same time, the transition probability pi,j for 1 ≤ i ≤ 5, −3 ≤ j ≤ 0, is the multiplication of Fi *prl_i , where Fi is the failure probability from any state i (1 ≤ i ≤ 5) to below standard condition -3,-2,-1, 0. prl_k is the probability of the repair length (rl): k= 1,2,3,4. In matrix notion, we have: Fi = ( 0.01 0.02 0.08 0.15 0.3 ); prl_i = (0.45 0.25 0.20 0.10); Hence, the transition probability Fi*prl_i gives us the probability of repair length for each equipment’s working conditions.
  • 22. Examination Number: B025068 22 pi,j -3 -2 -1 0 1 0.01 0.02 0.025 0.0045 2 0.002 0.004 0.005 0.009 3 0.008 0.016 0.02 0.036 4 0.0015 0.03 0.0375 0.068 5 0.03 0.06 0.075 0.135 In addition, under the preventive maintenance, the working condition improves from i to the nearest upper condition i-1, except for conditions 1 and 2, that is pi,i−1 = 1 for i = 3,4,5; p1,1 = 1; p2,1 = 0.5. Furthermore, the condition after the repair is 1 for all. Optimal Equation: The main problem that we consider in this project is to find Vi, the maximum expected discounted reward over an infinite horizon when the process is in state i initially. The optimality equation is: 𝑣𝑖 𝑛 = 𝑚𝑎𝑥 𝑘 ∊ 𝐾𝑖 {𝑟𝑖 𝑘 + 𝛽(𝑝𝑖,1 𝑘 𝑣1 𝑛−1 + 𝑝𝑖,2 𝑘 𝑣2 𝑛−1 + ⋯ + 𝑝𝑖,𝑀 𝑘 𝑣 𝑀 𝑛−1 )}
  • 23. Examination Number: B025068 23 For Example: Formula used for v_n,-3,re is =r__3re_A+df*SUMPRODUCT(p__3re,B3:J3) AH4=ABS(B4-B3) F4=MAX(X4:Y4)
  • 24. Examination Number: B025068 24 Immediate reward and transition probability for Part A and Part B only (Without replacement) Appendix ii: Value Iteration Results for Scenario A & B & C (Beginning of the calculation and a few at the end of the calculation) Scenario A:
  • 33. Examination Number: B025068 33 Appendix iii: Proportion of Time “operate” and “preventive maintenance” For Scenario A & B : The result is Thus, according to the decision of the model, we can list the transition probability for A&B with the optimal decision 𝒑𝒊,𝒋 -3 -2 -1 0 1 2 3 4 5 -3 0 1 0 0 0 0 0 0 0 CONDITION -3 -2 -1 0 1 2 3 4 5 OPTIMAL POLICY OF A repair repair repair repair Operate Operate Operate Operate Operate OPTIMAL POLICY OF B repair repair repair repair Operate Operate Operate Operate Operate
  • 34. Examination Number: B025068 34 -2 0 0 1 0 0 0 0 0 0 -1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0.001 0.002 0.0025 0.0045 0.95 0.03 0.01 0 0 2 0.002 0.004 0.005 0.009 0 0.9 0.05 0.02 0.01 3 0.008 0.016 0.02 0.036 0 0 0.8 0.08 0.04 4 0.015 0.03 0.0375 0.0675 0 0 0 0.7 0.15 5 0.03 0.06 0.075 0.135 0 0 0 0 0.7 Then, the Global Balance Equation is 𝝅−𝟑 = 𝟎. 𝟎𝟎𝟏𝝅 𝟏 + 𝟎. 𝟎𝟎𝟐𝝅 𝟐 + 𝟎. 𝟎𝟎𝟖𝝅 𝟑 + 𝟎. 𝟎𝟏𝟓𝝅 𝟒 + 𝟎. 𝟎𝟑𝝅 𝟓 𝝅−𝟐 = 𝟎. 𝟎𝟎𝟐𝝅 𝟏 + 𝟎. 𝟎𝟎𝟒𝝅 𝟐 + 𝟎. 𝟎𝟎𝟏𝟔𝝅 𝟑 + 𝟎. 𝟎𝟑𝝅 𝟒 + 𝟎. 𝟎𝟔𝝅 𝟓 + 𝝅−𝟑 𝝅−𝟏 = 𝟎. 𝟎𝟎𝟐𝟓𝝅 𝟏 + 𝟎. 𝟎𝟎𝟓𝝅 𝟐 + 𝟎. 𝟎𝟎𝟐𝝅 𝟑 + 𝟎. 𝟎𝟑𝟕𝟓𝝅 𝟒 + 𝟎. 𝟎𝟕𝟓𝝅 𝟓 + 𝝅−𝟐 𝝅 𝟎 = 𝟎. 𝟎𝟎𝟒𝟓𝝅 𝟏 + 𝟎. 𝟎𝟗𝝅 𝟐 + 𝟎. 𝟎𝟎𝟑𝟔𝝅 𝟑 + 𝟎. 𝟎𝟔𝟕𝟓𝝅 𝟒 + 𝟎. 𝟏𝟑𝟓𝝅 𝟓 + 𝝅−𝟏 𝝅 𝟏 = 𝝅 𝟎 + 𝟎. 𝟗𝟓𝝅 𝟏 𝝅 𝟐 = 𝟎. 𝟎𝟑𝝅 𝟏 + 𝟎. 𝟗𝝅 𝟐 𝝅 𝟑 = 𝟎. 𝟎𝟏𝝅 𝟏 + 𝟎. 𝟎𝟓𝝅 𝟐 + 𝟎. 𝟖𝝅 𝟑 𝝅 𝟒 = 𝟎. 𝟎𝟐𝝅 𝟏 + 𝟎. 𝟎𝟖𝝅 𝟐 + 𝟎. 𝟕𝝅 𝟑 𝝅 𝟓 = 𝟎. 𝟎𝟎𝟏𝝅 𝟐 + 𝟎. 𝟎𝟒𝝅 𝟑 + 𝟎. 𝟏𝟓𝝅 𝟒 + 𝟎. 𝟕𝝅 𝟓 𝝅−𝟑+𝝅−𝟐 + 𝝅−𝟏 + 𝝅 𝟎 + 𝝅 𝟏 + 𝝅 𝟐 + 𝝅 𝟑 + 𝝅 𝟒 + 𝝅 𝟓 = 𝟏 Totally, the mathematical result for A & B : 𝝅−𝟏 = 𝟎. 𝟏𝟐𝟓𝟔𝟗𝟑; 𝝅−𝟐 = 𝟎. 𝟏𝟏𝟖𝟐𝟐𝟕; 𝝅−𝟑 = 𝟎. 𝟎𝟎𝟐𝟗𝟖𝟐; 𝝅 𝟎 = 𝟎. 𝟎𝟐𝟐𝟗 𝝅 𝟏 = 𝟎. 𝟒𝟓𝟕𝟗𝟗𝟔; 𝝅 𝟐 = 𝟎. 𝟏𝟑𝟕𝟑𝟗𝟗; 𝝅 𝟑 = 𝟎. 𝟎𝟒𝟓𝟖; 𝝅 𝟒 = 𝟎. 𝟎𝟓𝟐𝟐𝟏𝟐; 𝝅 𝟓 = 𝟎. 𝟎𝟑𝟔𝟕𝟗𝟐 The proportion of time to be “operate” is 𝝅 𝟏 + 𝝅 𝟐 + 𝝅 𝟑 + 𝝅 𝟒 + 𝝅 𝟓 = 𝟎. 𝟕𝟑𝟎𝟏𝟗𝟗=73.02%
  • 35. Examination Number: B025068 35 The proportion of time to be “out of use” is 𝝅−𝟏 + 𝝅−𝟐 + 𝝅−𝟑 + 𝝅 𝟎 = 𝟎. 𝟐𝟔𝟗𝟖𝟎𝟐=26.98% For Scenario C : The result is Thus, according to the decision of the model, we can list the transition probability for C with the optimal decision 𝒑𝒊,𝒋 -3 -2 -1 0 1 2 3 4 5 -3 0 1 0 0 0 0 0 0 0 -2 0 0 1 0 0 0 0 0 0 -1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0.001 0.002 0.0025 0.0045 0.95 0.03 0.01 0 0 2 0 0 0 0 0.5 0.5 0 0 0 3 0 0 0 0 0 1 0 0 0 4 0.015 0.03 0.0375 0.0675 0 0 0 0.7 0.15 5 0.03 0.06 0.075 0.135 0 0 0 0 0.7 Then, the Global Balance Equation is 𝝅−𝟑 = 𝟎. 𝟎𝟎𝟏𝝅 𝟏 + 𝟎. 𝟎𝟏𝟓𝝅 𝟒 + 𝟎. 𝟎𝟑𝝅 𝟓 𝝅−𝟐 = 𝟎. 𝟎𝟎𝟐𝝅 𝟏 + 𝟎. 𝟎𝟑𝝅 𝟒 + 𝟎. 𝟎𝟔𝝅 𝟓 + 𝝅−𝟑 𝝅−𝟏 = 𝟎. 𝟎𝟎𝟐𝟓𝝅 𝟏 + 𝟎. 𝟎𝟑𝟕𝟓𝝅 𝟒 + 𝟎. 𝟎𝟕𝟓𝝅 𝟓 + 𝝅−𝟐 𝝅 𝟎 = 𝟎. 𝟎𝟎𝟒𝟓𝝅 𝟏 + 𝟎. 𝟎𝟔𝟕𝟓𝝅 𝟒 + 𝟎. 𝟏𝟑𝟓𝝅 𝟓 + 𝝅−𝟏 𝝅 𝟏 = 𝝅 𝟎 + 𝟎. 𝟗𝟓𝝅 𝟏 + 𝟎. 𝟓𝝅 𝟐 CONDITIO N -3 -2 -1 0 1 2 3 4 5 OPTIMAL POLICY OF A repai r repai r repai r repai r Operat e Preventive Maintenanc e Preventive Maintenanc e Operat e Operat e
  • 36. Examination Number: B025068 36 𝝅 𝟐 = 𝟎. 𝟎𝟑𝝅 𝟏 + 𝟎. 𝟓𝝅 𝟐 𝝅 𝟑 𝝅 𝟑 = 𝟎. 𝟎𝟏𝝅 𝟏 𝝅 𝟒 = 𝟎. 𝟕𝝅 𝟒 𝝅 𝟓 = 𝟎. 𝟏𝟓𝝅 𝟒 + 𝟎. 𝟕𝝅 𝟓 𝝅−𝟑+𝝅−𝟐 + 𝝅−𝟏 + 𝝅 𝟎 + 𝝅 𝟏 + 𝝅 𝟐 + 𝝅 𝟑 + 𝝅 𝟒 + 𝝅 𝟓 = 𝟏 Totally, the mathematical result for A & B : 𝝅−𝟏 = 𝟎. 𝟎𝟏𝟐𝟕𝟓𝟖𝟒𝟎𝟓; 𝝅−𝟐 = 𝟎. 𝟎𝟏𝟎𝟓𝟐𝟖𝟔𝟖; 𝝅−𝟑 = 𝟎. 𝟎𝟎𝟎𝟖𝟕𝟗𝟖𝟗; 𝝅 𝟎 = 𝟎. 𝟎𝟏𝟔𝟕𝟏𝟗𝟕𝟏 𝝅 𝟏 = 𝟎. 𝟖𝟕𝟗𝟖𝟗; 𝝅 𝟐 = 𝟎. 𝟎𝟕𝟎𝟑𝟗𝟏𝟓𝟓; 𝝅 𝟑 = 𝟎. 𝟎𝟎𝟖𝟕𝟗𝟖𝟗; 𝝅 𝟒 = 𝝅 𝟓 = 𝟎 The proportion of time to be “operate” is 𝝅 𝟏 + 𝝅 𝟒 + 𝝅 𝟓 =0.87989=87.99% The proportion of time to be “out of use” is 𝝅−𝟏 + 𝝅−𝟐 + 𝝅−𝟑 + 𝝅 𝟎 + 𝝅 𝟐 + 𝝅 𝟑 =0.12011=12.01% Appendix iv: Sensitivity Analysis  Sensitivity Analysis of repair length of A&B&C for day 1,2,3,4.
  • 37. Examination Number: B025068 37  Sensitivity Analysis for repair costs for A,B,C between £100-£1000
  • 38. Examination Number: B025068 38  Sensitivity Analysis of Discount Factor of A  Sensitivity Analysis of Preventive Costs:
  • 39. Examination Number: B025068 39 Appendix V: Replacement Results and Sensitivity Analysis (Part C)