This presentation presents techniques and experimental results for considering prediction reliability and risk during predictive business process monitoring. Considering reliability and risk provides additional decision support for proactive process adaptation.

1. Predictive Business Process
Monitoring unter Berücksichtigung
von Prognoseverlässlichkeit und
Risiko
Andreas Metzger, Philipp Bohn, Felix Föcker
CAiSE 2017 - https://doi.org/10.1007/978-3-319-59536-8_28
ICSOC 2017 - https://doi.org/10.1007/978-3-319-69035-3_25

2. Motivation
Predictive Monitoring and Proactive Adaptation
2SE 2018, Ulm
monitor
predict
real-time
decision
proactive
adaptation
time
t t + 
planned /
acceptable situations
= Violation
= Non-
Violation

e.g., delay in
freight delivery
time
e.g., schedule
faster means of
transport

3. Agenda
1. Considering Reliability
2. Considering Risk
3. Conclusions and Perspectives
SE 2018, Ulm 3

4. Considering Reliability
Prediction Accuracy
SE 2018, Ulm 4
• Prediction accuracy is key for proactive process adaptation
• Prediction accuracy = ability of prediction technique
• to forecast as many true violations as possible,
• while generating as few false alarms as possible
• True violation  triggering of required adaptations
• Missed required adaptation = less opportunity for proactively preventing
or mitigating a problem
• False alarm  triggering of unnecessary adaptation
• Unnecessary adaptation = additional costs for executing the adaptations,
while not addressing actual problems

5. Considering Reliability
Prediction Accuracy
• Research focused on aggregate accuracy
• E.g., precision, recall, mean average prediction error, …
• But: aggregate accuracy gives no direct information about error of an
individual prediction
• Prediction reliability estimates provide such information
SE 2018, Ulm 5
Aggregate Accuracy
75%
75%
75%
75%
 Distinguish between more or less reliable predictions on case by case basis
Prediction #
1
2
3
…
Reliability Estimate
60%
90%
70%
…

6. Considering Reliability
Predictive Monitoring with Reliability Estimates
SE 2018, Ulm 6
monitor
predict
real-time
decision
time
t t + 
planned /
acceptable situations
= Violation
= Non-
Violation

 ≤ threshold  no adaptation
 > threshold  adaptation
+ Reliability estimate 
Reliability estimates offer more information for decision making
proactive
adaptation

7. Considering Reliability
Computing Predictions and Reliability Estimates
Foundation: Ensemble prediction using Machine Learning
SE 2018, Ulm 7
Prediction T
Reliability estimate 
Process
Monitoring
Data
Classification Model 1
Classification Model m{
{{ Each model of ensemble
trained differently
(bagging)
 T1
 Tm

8. Considering Reliability
Experimental Design
Cost Model and Experimental Variables
SE 2018, Ulm 8
Costs
Adaptation Cost
Adaptation Cost
+ Penalty
 > 
 ≤  No
Adaptation
Adaptation
Reliability 
Violation
Non-Violationeffective
not
effective
0
PenaltyViolation
Non-Violation
• Reliability threshold 
• Adaptation effectiveness 
• Relative adaptation costs : adaptation cost =  · penalty

9. Considering Reliability
Experimental Design
Process Model and Data Set
Airfreight process
• 5 months of operational data
• 3 942 process instances
• 56 082 service invocations
9SE 2018, Ulm
Point of
Prediction

10. Considering Reliability
Experimental Results
10SE 2018, Ulm
EnsSize=100, BootsSize=80, Voting=simple, AdaptSuccRate=0.80
Tot - Lambda [%] 0 Tot - Lambda [%] 10 Tot - Lambda [%] 20 Tot - Lambda [%] 30
Tot - Lambda [%] 40 Tot - Lambda [%] 50 Tot - Lambda [%] 60 Tot - Lambda [%] 70
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0,80 0,85 0,90 0,95 1,00 1,05
Reliability Threshold
60.000
70.000
80.000
90.000
100.000
110.000
120.000
130.000
Cost
 (reliability threshold)
costs
 = .9
 = .8
 = 1
 = .7
 = .6
 = .5
 = .4
 = .3
 = .2
 = 0
 = .1
.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.0






130,000
120,000
110,000
100,000
90,000
80,000
70,000
60,000
No proactive
process adaptationProactive process adaptation w/out reliability
Key:
Negative effect
Positive effect
 Optimum
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
0,88
NonViolationRate
alpha-0.1 alpha-0.2 alpha-0.3 alpha-0.4 alpha-0
alpha-0.8000001 alpha-0.9000001 alpha-1.0000001
0,45 0,50 0,55 0,60 0,65 0,70 0,75 0
Reliability Thresh
0,72
0,74
0,76
0,78
0,8
0,82
0,84
0,86
NonViolationRate
 = .8
Very high adaptation
costs, not compensated
by avoided penalties
Missed required adaptations
(even though cheap) due to
“loss” of predictions

11. Considering Reliability
Experimental Results
• Observations for full range of , ,  (= 5000 cases)
• Striving balance between avoiding unnecessary proactive
actions and rejecting required proactive actions
• Cost savings due to proactive process adaptation
• No, in 47.5% of the cases
• Yes, in 52.5% of the cases
• Cost savings due to considering
reliability estimates
• No, in 17,1% of the cases
• Yes, in 82.9% of the cases
12SE 2018, Ulm
Cost savings
Frequency
Savings from 2%
to 54%,
14% on average

12. Agenda
1. Considering Reliability
2. Considering Risk
3. Conclusions and Perspectives
SE 2018, Ulm 13

13. Accuracy of individual prediction
 Reliability estimate to quantify probability of violation
Severity of violation
• E.g., contractual penalties (such as stipulated in SLAs)
 Estimated penalty to quantify severity (in terms of costs)
 Risk = Probability of occurrence × Severity [ISO 31000:2009]
 Risk = Reliability estimate × Estimated penalty
SE 2018, Ulm 14
Considering Risk
Factors impacting the success of the adaptation decision

14. Considering Risk
Risk-based Proactive Process Adaptation Decision
SE 2018, Ulm 15
monitor
predict
real-time
decision
proactive
adaptation
time
t t + 
planned /
acceptable situations
= Violation
= Non-
Violation

R ≤ threshold  no adaptation
R > threshold  adaptation
+ Risk R
Risk estimate as basis for decision making

15. Ensemble
Considering Risk
Computing Penalty
16SE 2018, Ulm
 Penalty =
Process
Monitoring
Data
{  =
1
𝑛
×
𝑖=1,…,𝑛
𝑎𝑖 − 𝐴
Regression Model 1
Regression Model n
 a1
 an
{
Deviation
δ
Linear with cap
clin
c
0 1
δ
Constantc
0
cconst
1
c
Step-wise (s steps)
δ
1/s 2/s
cstep
1
2/s·cstep
1/s·cstep
(s-1)/s
…
0
𝑐()

16. Considering Risk
Experimental Results
Penalty R = 0.1 R = 0.3 R = 0.5 R = 0.7 R = 0.9
constant -19.0 -20.0 -17.0 -3.0 3.1
step-wise -14.0 12.0 20.0 20.0 8.6
linear 0.6 21.0 27.0 26.0 11.0
SE 2018, Ulm 17
Cost savings (averaged over α ={0.1, 0.2, 0.3, … ,1})
Constant penalty Step-wise penalty Linear penalty
Risk threshold R
Costsavings

17. Conclusions and Perspectives
Predictive business process monitoring
• Prediction of potential problems before they occur
• Proactive adaptation of processes
• Cost Savings
• Reliability: 14% on average
• Risk: + 23% on average
Deep Learning for Process Prediction
• Recurrent Neural Networks (RNNs) with LSTM
• Initial results: 27% higher accuracy than Multi-Layer Perceptron (MLP)
SE 2018, Ulm 18
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0 10 20 30 40 50 60 70 80 90
Diagrammtitel
num s2e noplannednopath mlp
Checkpoint [relative prefix]
Accuracy[MCC]
RNN
MLP

18. Thank You / Danke
SE 2018, Ulm 19
…the EFRE co-financed operational
program NRW.Ziel2
http://www.lofip.de
…the EU’s Horizon 2020 research and
innovation programme under Objective
ICT-15 ‘Big Data PPP: Large Scale Pilot
Actions ‘
http://www.transformingtransport.eu
Research leading to these results has received
funding from…