Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore
Can I Trust My Simulation
Model? Measuring the
Quality of Business Process
Simulation Models
David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2,
Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3
21st International Conference on Business Process
Management (BPM 2023)
1 University of Tartu, Estonia
2 University of Toronto, Canada
3 York University, Canada
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2
Introduction
Business Process Simulation (BPS)
3
BPS allows users to address “what-if” analysis questions.
What would be the cycle time of the process if the rate of arrival of new cases
doubles?
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS)
4
BPS models can be manually created by modeling experts.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS)
5
BPS models can be manually created by modeling experts.
Use of process mining techniques to automatically discover BPS
models from business process event logs.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS)
6
How to assess the quality of a BPS model?
Automatic assessment.
Useful to detect the sources of deviations.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7
Proposed Framework
Quality of a BPS model
8
How to assess the quality of a BPS model?
Comparing an event log with a BPS model.
Variety of different BPS models formats.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log
Quality of a BPS model
9
How to assess the quality of a BPS model?
Generate K simulated event logs.
Compare individually and report the average and confidence interval.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
9
Process event log
K simulated event logs
Quality of a BPS model
10
A BPS model can be very accurate in one aspect (e.g., control-flow), yet
very different in another (e.g., processing times).
Three main dimensions: control-flow, temporal, congestion.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11
Proposed Framework
Control-flow measures
Control-flow: Control-Flow Log
Distance
12
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs:
data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021)
Process event log Simulated event log
Control-flow: Control-Flow Log
Distance
13
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
Control-flow: Control-Flow Log
Distance
14
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
0
Control-flow: Control-Flow Log
Distance
15
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
0
0
0.2
Control-flow: Control-Flow Log
Distance
16
Control-Flow Log Distance (CFLD): given two event logs L1 and L2,
(minimum) average distance to transform each case in L1 into another
case in L2, such that each case in L1 is paired to a different case in L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
CFLD =
0+0+0.75+0+0.2
5
= 0.19
Control-flow: N-Gram Distance
17
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance
checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019)
Process event log Simulated event log
Control-flow: N-Gram Distance
18
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
19
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
20
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
21
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
A B C D
A B C D
A C B D
A E F G H
A E F G I
A B C D
A C B D
A E F G
A E F G H
A E F G H
N = 3
Control-flow: N-Gram Distance
22
N-Gram Distance (NGD): given two event logs L1 and L2, and a positive
integer 𝑛, difference in the frequencies of the 𝑛-grams observed in
both L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
0
1
2
3
4
5
6
_ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _
Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23
Proposed Framework
Temporal measures
Process event log Simulated event log
Temporal: Absolute Event
Distribution
24
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log
Temporal: Absolute Event
Distribution
25
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Temporal: Absolute Event
Distribution
26
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
06-10-2022 10am – 11am
07-10-2022 11am – 12pm
Temporal: Absolute Event
Distribution
27
Absolute Event Distribution (AED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2.
How different they are distributed through the event log.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Earth
mover's
distance
Temporal: Circadian Event
Distribution
28
Circadian Event Distribution (CED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, for each
day of the week.
How different they are distributed through each day of the week.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Monday
Tuesday
Wednesday
Thursday 10am – 11am
Friday 11am – 12pm
Temporal: Circadian Event
Distribution
29
Circadian Event Distribution (CED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, for each
day of the week.
How different they are distributed through each day of the week.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Monday Monday
Temporal: Relative Event
Distribution
30
Relative Event Distribution (RED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, with
respect to the start of their case.
How different they are distributed within each process case.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
00:00:00 01:01:47
Temporal: Relative Event
Distribution
Relative Event Distribution (RED): given two event logs L1 and L2,
distance between the time series of the events in L1 and L2, with
respect to the start of their case.
How different they are distributed within each process case.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32
Proposed Framework
Congestion measures
Congestion: Case Arrival Rate
33
Case Arrival Rate (CAR): given two event logs L1 and L2, distance
between how the case arrivals are distributed in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
06-10-2022 10am – 11am
Congestion: Case Arrival Rate
34
Case Arrival Rate (CAR): given two event logs L1 and L2, distance
between how the case arrivals are distributed in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Congestion: Cycle Time Distribution
35
Cycle Time Distribution (CTD): given two event logs L1 and L2, distance
between the distribution of cycle times in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
01:07:02
Congestion: Cycle Time Distribution
36
Cycle Time Distribution (CTD): given two event logs L1 and L2, distance
between the distribution of cycle times in L1 and L2.
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37
Evaluation
Evaluation
38
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ1: Are the proposed measures able to discern the impact of different
known modifications to a BPS model?
EQ2: Is the N-Gram Distance’s performance significantly different from
the CFLD’s performance?
No modifications
Control-flow
Gateway probabilities
Case arrival rate
Activity durations
Resource contention
Working calendars
Extraneous delays
Evaluation
39
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Evaluation
40
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ1: Are the proposed measures able to discern the impact of different
known modifications to a BPS model?
Evaluation
41
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ2: Is the N-Gram Distance’s performance significantly different from
the CFLD’s performance?
Kendall
rank
correlation
coefficient
1.0
Evaluation
42
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
43
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
4 real-life processes: each split into disjoint training and test.
Evaluation
44
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Automatically discover BPS model with SIMOD and Service Miner.
Evaluation
45
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
46
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
47
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
48
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ3: Given two BPS models discovered by existing automated BPS
model discovery techniques in real-life scenarios, are the proposed
measures able to identify the strengths and weaknesses of each
technique?
Evaluation
49
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
EQ4: Does the 1-WD report the same insights in real-life scenarios as
the EMD?
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50
Conclusion
Conclusion
51
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Proposed a framework to measure the quality of a BPS model:
decomposing into three perspectives (control-flow, temporal, and
congestion), and defined measures for each of these perspectives.
The measures proved their ability to detect the alterations in their
corresponding perspectives.
Beyond capturing the quality of BPS model and identifying the sources of
discrepancies, the measures can also assist in eliciting areas for
improvement in these techniques.
The presented computationally efficient alternatives led to similar
conclusions.
Future Work
52
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Explore the applicability of the proposed measures to other process
mining problems, e.g., concept drift detection and variant analysis.
Studying how to assess the quality of BPS models in the context of
object-centric event logs.
Study other quality measures for BPS models adapted from the field of
generative machine learning, for example, by using a discriminative
model that attempts to distinguish between data generated by the
BPS model and real data.
1 of 52

Recommended

Walking the Way from Process Mining to AI-Driven Process Optimization by
Walking the Way from Process Mining to AI-Driven Process OptimizationWalking the Way from Process Mining to AI-Driven Process Optimization
Walking the Way from Process Mining to AI-Driven Process OptimizationMarlon Dumas
672 views27 slides
Business Process Optimization: Status and Perspectives by
Business Process Optimization: Status and PerspectivesBusiness Process Optimization: Status and Perspectives
Business Process Optimization: Status and PerspectivesMarlon Dumas
544 views18 slides
Fundamentals of Business Process Management - Tutorial at CAiSE'2018 by
Fundamentals of Business Process Management - Tutorial at CAiSE'2018Fundamentals of Business Process Management - Tutorial at CAiSE'2018
Fundamentals of Business Process Management - Tutorial at CAiSE'2018Marlon Dumas
6K views60 slides
Process architecture - Part II by
Process architecture - Part IIProcess architecture - Part II
Process architecture - Part IIMarcello La Rosa
12.7K views30 slides
Building Event-Driven (Micro) Services with Apache Kafka by
Building Event-Driven (Micro) Services with Apache KafkaBuilding Event-Driven (Micro) Services with Apache Kafka
Building Event-Driven (Micro) Services with Apache KafkaGuido Schmutz
1.4K views47 slides
Visualizing Software Architecture with C4 Model by
Visualizing Software Architecture with C4 ModelVisualizing Software Architecture with C4 Model
Visualizing Software Architecture with C4 ModelMarco Beelen
1.2K views21 slides

More Related Content

What's hot

Apache Kafka® Use Cases for Financial Services by
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Servicesconfluent
4.1K views41 slides
Agile, User Stories, Domain Driven Design by
Agile, User Stories, Domain Driven DesignAgile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven DesignAraf Karsh Hamid
1.2K views192 slides
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture by
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureChris Richardson
3.8K views36 slides
Microservices Part 3 Service Mesh and Kafka by
Microservices Part 3 Service Mesh and KafkaMicroservices Part 3 Service Mesh and Kafka
Microservices Part 3 Service Mesh and KafkaAraf Karsh Hamid
7.6K views56 slides
Enterprise Architecture Toolkit Overview by
Enterprise Architecture Toolkit OverviewEnterprise Architecture Toolkit Overview
Enterprise Architecture Toolkit OverviewMike Walker
4.5K views29 slides
Microsoft DevOps Solution - DevOps by
Microsoft DevOps Solution - DevOps  Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps Chetan Gordhan
712 views34 slides

What's hot(20)

Apache Kafka® Use Cases for Financial Services by confluent
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
confluent4.1K views
Agile, User Stories, Domain Driven Design by Araf Karsh Hamid
Agile, User Stories, Domain Driven DesignAgile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven Design
Araf Karsh Hamid1.2K views
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture by Chris Richardson
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
Chris Richardson3.8K views
Microservices Part 3 Service Mesh and Kafka by Araf Karsh Hamid
Microservices Part 3 Service Mesh and KafkaMicroservices Part 3 Service Mesh and Kafka
Microservices Part 3 Service Mesh and Kafka
Araf Karsh Hamid7.6K views
Enterprise Architecture Toolkit Overview by Mike Walker
Enterprise Architecture Toolkit OverviewEnterprise Architecture Toolkit Overview
Enterprise Architecture Toolkit Overview
Mike Walker4.5K views
Microsoft DevOps Solution - DevOps by Chetan Gordhan
Microsoft DevOps Solution - DevOps  Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps
Chetan Gordhan712 views
CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom... by Bernd Ruecker
CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom...CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom...
CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom...
Bernd Ruecker396 views
Microservices Docker Kubernetes Istio Kanban DevOps SRE by Araf Karsh Hamid
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Araf Karsh Hamid4.2K views
Cloud Operating Model Design by Joseph Schwartz
Cloud Operating Model DesignCloud Operating Model Design
Cloud Operating Model Design
Joseph Schwartz10.1K views
Enterprise Architecture - Overview by Faqih Zulfikar
Enterprise Architecture - OverviewEnterprise Architecture - Overview
Enterprise Architecture - Overview
Faqih Zulfikar588 views
Event-Driven Architecture (EDA) by WSO2
Event-Driven Architecture (EDA)Event-Driven Architecture (EDA)
Event-Driven Architecture (EDA)
WSO210.9K views
Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro... by confluent
Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro...Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro...
Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro...
confluent3.7K views
Data Pipline Observability meetup by Omid Vahdaty
Data Pipline Observability meetup Data Pipline Observability meetup
Data Pipline Observability meetup
Omid Vahdaty371 views
MLOps journey at Swisscom: AI Use Cases, Architecture and Future Vision by BATbern
MLOps journey at Swisscom: AI Use Cases, Architecture and Future VisionMLOps journey at Swisscom: AI Use Cases, Architecture and Future Vision
MLOps journey at Swisscom: AI Use Cases, Architecture and Future Vision
BATbern220 views
Decomposing Applications for Scalability and Deployability (April 2012) by Chris Richardson
Decomposing Applications for Scalability and Deployability (April 2012)Decomposing Applications for Scalability and Deployability (April 2012)
Decomposing Applications for Scalability and Deployability (April 2012)
Chris Richardson18.2K views
SQL Analytics Powering Telemetry Analysis at Comcast by Databricks
SQL Analytics Powering Telemetry Analysis at ComcastSQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at Comcast
Databricks341 views
Application modernization patterns with apache kafka, debezium, and kubernete... by Bilgin Ibryam
Application modernization patterns with apache kafka, debezium, and kubernete...Application modernization patterns with apache kafka, debezium, and kubernete...
Application modernization patterns with apache kafka, debezium, and kubernete...
Bilgin Ibryam748 views

Similar to Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Process Capability by
Process CapabilityProcess Capability
Process CapabilityThomas Abraham
7K views41 slides
Process Mining and Predictive Process Monitoring by
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process MonitoringMarlon Dumas
3.3K views40 slides
Statistical Process Control - SPC by
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPCPrasenjit Puri
153 views25 slides
Six Sigma Dfss Application In Data Accarucy by
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucyxyhfun
529 views28 slides
Process Mining and Predictive Process Monitoring in Apromore by
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in ApromoreMarlon Dumas
713 views34 slides
A process to improve the accuracy of mk ii fp to cosmic charles symons by
A process to improve the accuracy of mk ii fp to cosmic    charles symonsA process to improve the accuracy of mk ii fp to cosmic    charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symonsIWSM Mensura
349 views30 slides

Similar to Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models(20)

Process Mining and Predictive Process Monitoring by Marlon Dumas
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process Monitoring
Marlon Dumas3.3K views
Statistical Process Control - SPC by Prasenjit Puri
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPC
Prasenjit Puri153 views
Six Sigma Dfss Application In Data Accarucy by xyhfun
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucy
xyhfun529 views
Process Mining and Predictive Process Monitoring in Apromore by Marlon Dumas
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in Apromore
Marlon Dumas713 views
A process to improve the accuracy of mk ii fp to cosmic charles symons by IWSM Mensura
A process to improve the accuracy of mk ii fp to cosmic    charles symonsA process to improve the accuracy of mk ii fp to cosmic    charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symons
IWSM Mensura349 views
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs by Marlon Dumas
Multi-Perspective Comparison of Business Processes Variants Based on Event LogsMulti-Perspective Comparison of Business Processes Variants Based on Event Logs
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs
Marlon Dumas557 views
Advantages Of Linear Programming Models by Brenda Torres
Advantages Of Linear Programming ModelsAdvantages Of Linear Programming Models
Advantages Of Linear Programming Models
Brenda Torres3 views
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op... by IRJET Journal
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
IRJET Journal41 views
Comprehensive acquisition management methodology rev6 by clifpt
Comprehensive acquisition  management methodology   rev6Comprehensive acquisition  management methodology   rev6
Comprehensive acquisition management methodology rev6
clifpt44 views
Gestión de la calidad sem 2 by youffre
Gestión de la calidad sem 2Gestión de la calidad sem 2
Gestión de la calidad sem 2
youffre118 views
Practical Guides on Programming with Big Number Library in Scientific Researches by theijes
Practical Guides on Programming with Big Number Library in Scientific ResearchesPractical Guides on Programming with Big Number Library in Scientific Researches
Practical Guides on Programming with Big Number Library in Scientific Researches
theijes31 views
Unbiased, Fine-Grained Description of Processes Performance from Event Data by Vadim Denisov
Unbiased, Fine-Grained Description of Processes Performance from Event DataUnbiased, Fine-Grained Description of Processes Performance from Event Data
Unbiased, Fine-Grained Description of Processes Performance from Event Data
Vadim Denisov34 views
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING by IJCI JOURNAL
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKINGBUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
IJCI JOURNAL2 views
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN... by IAEME Publication
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
IAEME Publication267 views
Decisions in a supply chain modeling for comparative evaluation strategies in... by IAEME Publication
Decisions in a supply chain modeling for comparative evaluation strategies in...Decisions in a supply chain modeling for comparative evaluation strategies in...
Decisions in a supply chain modeling for comparative evaluation strategies in...
IAEME Publication294 views
Software Architecture in the age of Cloud Computing by Jaroslav Gergic
Software Architecture in the age of Cloud ComputingSoftware Architecture in the age of Cloud Computing
Software Architecture in the age of Cloud Computing
Jaroslav Gergic525 views

More from Marlon Dumas

Discovery and Simulation of Business Processes with Probabilistic Resource Av... by
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Marlon Dumas
63 views15 slides
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... by
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Marlon Dumas
35 views19 slides
Augmented Business Process Management by
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process ManagementMarlon Dumas
1.2K views23 slides
Process Mining and Data-Driven Process Simulation by
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process SimulationMarlon Dumas
299 views29 slides
Modeling Extraneous Activity Delays in Business Process Simulation by
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process SimulationMarlon Dumas
148 views42 slides
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints by
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsMarlon Dumas
92 views23 slides

More from Marlon Dumas(20)

Discovery and Simulation of Business Processes with Probabilistic Resource Av... by Marlon Dumas
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Marlon Dumas63 views
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... by Marlon Dumas
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Marlon Dumas35 views
Augmented Business Process Management by Marlon Dumas
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process Management
Marlon Dumas1.2K views
Process Mining and Data-Driven Process Simulation by Marlon Dumas
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process Simulation
Marlon Dumas299 views
Modeling Extraneous Activity Delays in Business Process Simulation by Marlon Dumas
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process Simulation
Marlon Dumas148 views
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints by Marlon Dumas
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints
Marlon Dumas92 views
Robotic Process Mining by Marlon Dumas
Robotic Process MiningRobotic Process Mining
Robotic Process Mining
Marlon Dumas333 views
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable? by Marlon Dumas
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Marlon Dumas51 views
Learning Accurate Business Process Simulation Models from Event Logs via Auto... by Marlon Dumas
Learning Accurate Business Process Simulation Models from Event Logs via Auto...Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Marlon Dumas421 views
Process Mining: A Guide for Practitioners by Marlon Dumas
Process Mining: A Guide for PractitionersProcess Mining: A Guide for Practitioners
Process Mining: A Guide for Practitioners
Marlon Dumas141 views
Optimización de procesos basada en datos by Marlon Dumas
Optimización de procesos basada en datosOptimización de procesos basada en datos
Optimización de procesos basada en datos
Marlon Dumas386 views
Process Mining and AI for Continuous Process Improvement by Marlon Dumas
Process Mining and AI for Continuous Process ImprovementProcess Mining and AI for Continuous Process Improvement
Process Mining and AI for Continuous Process Improvement
Marlon Dumas207 views
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction by Marlon Dumas
Prescriptive Process Monitoring for Cost-Aware Cycle Time ReductionPrescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Marlon Dumas362 views
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio... by Marlon Dumas
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Marlon Dumas654 views
On the Road to AI-Infused Process Execution by Marlon Dumas
On the Road to AI-Infused Process ExecutionOn the Road to AI-Infused Process Execution
On the Road to AI-Infused Process Execution
Marlon Dumas270 views
Process Mining 2.0: From Insights to Actions by Marlon Dumas
Process Mining 2.0: From Insights to ActionsProcess Mining 2.0: From Insights to Actions
Process Mining 2.0: From Insights to Actions
Marlon Dumas742 views
Process Mining in Action: Self-service data science for business teams by Marlon Dumas
Process Mining in Action: Self-service data science for business teamsProcess Mining in Action: Self-service data science for business teams
Process Mining in Action: Self-service data science for business teams
Marlon Dumas407 views
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E... by Marlon Dumas
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Marlon Dumas437 views
Identifying Candidate Routines for Robotic Process Automation From Unsegmente... by Marlon Dumas
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Marlon Dumas444 views

Recently uploaded

PRIVACY AWRE PERSONAL DATA STORAGE by
PRIVACY AWRE PERSONAL DATA STORAGEPRIVACY AWRE PERSONAL DATA STORAGE
PRIVACY AWRE PERSONAL DATA STORAGEantony420421
7 views56 slides
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation by
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented GenerationDataScienceConferenc1
19 views29 slides
Construction Accidents & Injuries by
Construction Accidents & InjuriesConstruction Accidents & Injuries
Construction Accidents & InjuriesBisnar Chase Personal Injury Attorneys
6 views5 slides
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ... by
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...DataScienceConferenc1
5 views19 slides
Dr. Ousmane Badiane-2023 ReSAKSS Conference by
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceDr. Ousmane Badiane-2023 ReSAKSS Conference
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceAKADEMIYA2063
5 views34 slides
DGST Methodology Presentation.pdf by
DGST Methodology Presentation.pdfDGST Methodology Presentation.pdf
DGST Methodology Presentation.pdfmaddierlegum
7 views9 slides

Recently uploaded(20)

PRIVACY AWRE PERSONAL DATA STORAGE by antony420421
PRIVACY AWRE PERSONAL DATA STORAGEPRIVACY AWRE PERSONAL DATA STORAGE
PRIVACY AWRE PERSONAL DATA STORAGE
antony4204217 views
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation by DataScienceConferenc1
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ... by DataScienceConferenc1
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
Dr. Ousmane Badiane-2023 ReSAKSS Conference by AKADEMIYA2063
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceDr. Ousmane Badiane-2023 ReSAKSS Conference
Dr. Ousmane Badiane-2023 ReSAKSS Conference
AKADEMIYA20635 views
DGST Methodology Presentation.pdf by maddierlegum
DGST Methodology Presentation.pdfDGST Methodology Presentation.pdf
DGST Methodology Presentation.pdf
maddierlegum7 views
LIVE OAK MEMORIAL PARK.pptx by ms2332always
LIVE OAK MEMORIAL PARK.pptxLIVE OAK MEMORIAL PARK.pptx
LIVE OAK MEMORIAL PARK.pptx
ms2332always7 views
4_4_WP_4_06_ND_Model.pptx by d6fmc6kwd4
4_4_WP_4_06_ND_Model.pptx4_4_WP_4_06_ND_Model.pptx
4_4_WP_4_06_ND_Model.pptx
d6fmc6kwd47 views
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P... by DataScienceConferenc1
[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
CRM stick or twist workshop by info828217
CRM stick or twist workshopCRM stick or twist workshop
CRM stick or twist workshop
info82821714 views
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo... by DataScienceConferenc1
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx by DataScienceConferenc1
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
Data about the sector workshop by info828217
Data about the sector workshopData about the sector workshop
Data about the sector workshop
info82821729 views
[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines by DataScienceConferenc1
[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines
[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines
Lack of communication among family.pptx by ahmed164023
Lack of communication among family.pptxLack of communication among family.pptx
Lack of communication among family.pptx
ahmed16402314 views
CRM stick or twist.pptx by info828217
CRM stick or twist.pptxCRM stick or twist.pptx
CRM stick or twist.pptx
info82821711 views

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

  • 1. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
  • 2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
  • 3. Business Process Simulation (BPS) 3 BPS allows users to address “what-if” analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 4. Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 5. Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 6. Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 7. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
  • 8. Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
  • 9. Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
  • 10. Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
  • 11. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
  • 12. Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
  • 13. Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
  • 14. Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
  • 15. Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
  • 16. Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
  • 17. Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019) Process event log Simulated event log
  • 18. Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 19. Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 20. Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 21. Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 22. Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
  • 23. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
  • 24. Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 25. Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 26. Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am 07-10-2022 11am – 12pm
  • 27. Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
  • 28. Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am – 11am Friday 11am – 12pm
  • 29. Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
  • 30. Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
  • 31. Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 32. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
  • 33. Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am
  • 34. Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 35. Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
  • 36. Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 37. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
  • 38. Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
  • 39. Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 40. Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
  • 41. Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? Kendall rank correlation coefficient 1.0
  • 42. Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 43. Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
  • 44. Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
  • 45. Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 46. Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 47. Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 48. Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 49. Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
  • 50. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
  • 51. Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
  • 52. Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.

Editor's Notes

  1. So, the first thing we need to know is, what is business process simulation?
  2. BPS aims to replicate the execution of a process, to mimic the behavior of the process, in a certain scenario (set of resources, etc.) analyzing its performance (KPIs) This allows users… The starting point is a BPS model… A process model annotated with a set of simulation parameters that define the scenario (resources, calendars, activity durations…). [NEXT]
  3. BPS models may be manually created based on information collected via interviews or empirical observations.. Or [NEXT…]
  4. they may be automatically discovered from execution data recorded in process-aware information systems (event logs) Regardless of the origin, a key question when using a BPS model is… [NEXT]
  5. how to assess its quality? Several approaches have been proposed to address this problem. However, these approaches are either manual and qualitative or they produce a single number that does not allow one to identify the source(s) of deviations between the BPS model and the observed reality
  6. First we need to decide what to compare when assessing the quality of a BPS model. What we are comparing is a BPS model, with a PROCESS What we usually have is…
  7. …an event log! Now, the first thing we asked ourselves was: should we compare a BPS model against a event log? But it is true that BPS models do not follow a standard structure… They can be formed by queue systems, but less of more models (resources, more complex waiting times), and they will change during time with new research. Thus, what we can do is simulate an event log out of the BPS model
  8. and compare log to log. K runs and compute the avg and conf int
  9. Abstract event logs into time-series or histograms and compare them
  10. We have two event logs, we are focusing on the control-flow, so the first step is to… [NEXT]
  11. obtain the activity sequences of each event log. Then, we compute the Damerau-Levenshtein (string edit distance) distance between each pair of cases… [NEXT]
  12. For exampe, [comment examples], we repeat this for each case Once we have all the pairings computed, we compute the matching between cases of one log to another (such as each case in one log is matched to one case in the other event log, with no repetitions… [NEXT]
  13. While minimizing the sum of distances using the Hungarian algorithm for optimal alignment. Finally, the CFLD measure is the average of these distances… [NEXT]
  14. The computational complexity of computing the DL-distance for all possible pairings is O(N2 ×MTL3) where N is the number of traces in the logs (assuming both logs have an equal number of cases, which holds in our setting) and MTL is the maximum trace length. Since all pairings are put into a matrix to compute the optimal alignment of cases (the one that minimizes the total sum of distances), CFLD’s memory complexity is quadratic on the number of cases. The optimal alignment of traces using the Hungarian algorithm has a cubic complexity on the number of cases.
  15. In the same way than for the CFLD, we are focusing on the control-flow, so the first step is to obtain the activity sequences of each event log. Leemans et al. measure the quality of a stochastic process model by mapping the model and a log to their Directly-Follows Graph (DFG), viewing each DFG as a histogram, and measuring the distance between these histograms. We note that the histogram of 2-grams of a log is equal to the histogram of its DFG. Given this observation, we generalize the approach of to n-grams, noting that the histogram of n-grams of a log is equal to the (n-1)th-Markovian abstraction of the log. Then… [NEXT]
  16. Let’s assume a size of N=3, so the N-grams are 3-grams (sequences of three activites). We compute all 3-grams observed in both logs, considering two dummy activities in the start and end of each trace. Then… [NEXT]
  17. We measure the frequency of each N-gram in each log…
  18. We measure the frequency of each N-gram in each log…
  19. And compute the sum of absolute differences between them, normalized by the sum of frequencies of all n-grams (value between 0-1). NGD is considerably more efficient than CFLD, as the construction of the histogram of n-grams is linear on the number of events in the log, and the same goes for computing the differences between the n-gram histograms.
  20. For the temporal measures, we first do the opposite of the control-flow, we abstract from the control-flow information… [NEXT]
  21. retaining only the events (in this case start and end)… [NEXT]
  22. Then we discretize these events into bins of 1h in the following way. Obtaining a time-series with the number of events happening in each hour of the process timeline.
  23. Once we have the temporal distribution (not a probabilistic distribution, but just the events occurring in the timeline), we compare both time-series with the EMD to measure the distance. We measure the trend.
  24. The same process is followed for the next measure, but in this case discretized to weekdays. In this way, we measure the seasonality of the events happening in the process.
  25. The temporal distribution of events of each day of the week is compared and then we compute the average distance of the 7 days.
  26. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  27. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  28. For the case arrival rate we want to measure how different the arrival of cases is. So, we retain only the events denoting the arrival of each case (start of first activity instance). Then we build the distribution in the same way than the previous metrics… [NEXT]
  29. and compare with EMD.