Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore
โ€ขโ€ข
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
Business Process Simulation (BPS) 3 BPS allows users to address โ€œwhat-ifโ€ analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth moversโ€™ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127โ€“143. Springer (2019) Process event log Simulated event log
Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am โ€“ 11am 07-10-2022 11am โ€“ 12pm
Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am โ€“ 11am Friday 11am โ€“ 12pm
Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am โ€“ 11am
Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distanceโ€™s performance significantly different from the CFLDโ€™s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distanceโ€™s performance significantly different from the CFLDโ€™s performance? Kendall rank correlation coefficient 1.0
Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.
1 of 52

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

โ€ขโ€ข
Download to read offline
Data & Analytics

Business Process Simulation (BPS) is an approach to analyze the performance of business processes under different scenarios. For example, BPS allows us to estimate what would be the cycle time of a process if one or more resources became unavailable. The starting point of BPS is a process model annotated with simulation parameters (a BPS model). BPS models may be manually designed, based on information collected from stakeholders and empirical observations, or automatically discovered from execution data. Regardless of its origin, a key question when using a BPS model is how to assess its quality. In this paper, we propose a collection of measures to evaluate the quality of a BPS model w.r.t. its ability to replicate the observed behavior of the process. We advocate an approach whereby different measures tackle different process perspectives. We evaluate the ability of the proposed measures to discern the impact of modifications to a BPS model, and their ability to uncover the relative strengths and weaknesses of two approaches for automated discovery of BPS models. The evaluation shows that the measures not only capture how close a BPS model is to the observed behavior, but they also help us to identify sources of discrepancies. Presentation delivered by David Chapela-Campa at the BPM'2023 conference, Utrecht, September 2023.

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore

Recommended

Walking the Way from Process Mining to AI-Driven Process Optimization by
Walking the Way from Process Mining to AI-Driven Process OptimizationWalking the Way from Process Mining to AI-Driven Process Optimization
Walking the Way from Process Mining to AI-Driven Process OptimizationMarlon Dumas
672 viewsโ€ข27 slides
Business Process Optimization: Status and Perspectives by
Business Process Optimization: Status and PerspectivesBusiness Process Optimization: Status and Perspectives
Business Process Optimization: Status and PerspectivesMarlon Dumas
544 viewsโ€ข18 slides
Fundamentals of Business Process Management - Tutorial at CAiSE'2018 by
Fundamentals of Business Process Management - Tutorial at CAiSE'2018Fundamentals of Business Process Management - Tutorial at CAiSE'2018
Fundamentals of Business Process Management - Tutorial at CAiSE'2018Marlon Dumas
6K viewsโ€ข60 slides
Process architecture - Part II by
Process architecture - Part IIProcess architecture - Part II
Process architecture - Part IIMarcello La Rosa
12.7K viewsโ€ข30 slides
Building Event-Driven (Micro) Services with Apache Kafka by
Building Event-Driven (Micro) Services with Apache KafkaBuilding Event-Driven (Micro) Services with Apache Kafka
Building Event-Driven (Micro) Services with Apache KafkaGuido Schmutz
1.4K viewsโ€ข47 slides
Visualizing Software Architecture with C4 Model by
Visualizing Software Architecture with C4 ModelVisualizing Software Architecture with C4 Model
Visualizing Software Architecture with C4 ModelMarco Beelen
1.2K viewsโ€ข21 slides

More Related Content

What's hot

Apache Kafkaยฎ Use Cases for Financial Services by
Apache Kafkaยฎ Use Cases for Financial ServicesApache Kafkaยฎ Use Cases for Financial Services
Apache Kafkaยฎ Use Cases for Financial Servicesconfluent
4.1K viewsโ€ข41 slides
Agile, User Stories, Domain Driven Design by
Agile, User Stories, Domain Driven DesignAgile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven DesignAraf Karsh Hamid
1.2K viewsโ€ข192 slides
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture by
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureChris Richardson
3.8K viewsโ€ข36 slides
Microservices Part 3 Service Mesh and Kafka by
Microservices Part 3 Service Mesh and KafkaMicroservices Part 3 Service Mesh and Kafka
Microservices Part 3 Service Mesh and KafkaAraf Karsh Hamid
7.6K viewsโ€ข56 slides
Enterprise Architecture Toolkit Overview by
Enterprise Architecture Toolkit OverviewEnterprise Architecture Toolkit Overview
Enterprise Architecture Toolkit OverviewMike Walker
4.5K viewsโ€ข29 slides
Microsoft DevOps Solution - DevOps by
Microsoft DevOps Solution - DevOps Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps Chetan Gordhan
712 viewsโ€ข34 slides

What's hot(20)

Apache Kafkaยฎ Use Cases for Financial Services by confluent
Apache Kafkaยฎ Use Cases for Financial ServicesApache Kafkaยฎ Use Cases for Financial Services
Apache Kafkaยฎ Use Cases for Financial Services
confluentโ€ข4.1K views
Agile, User Stories, Domain Driven Design by Araf Karsh Hamid
Agile, User Stories, Domain Driven DesignAgile, User Stories, Domain Driven Design
Agile, User Stories, Domain Driven Design
Araf Karsh Hamidโ€ข1.2K views
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture by Chris Richardson
QConPlus 2021: Minimizing Design Time Coupling in a Microservice ArchitectureQConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
QConPlus 2021: Minimizing Design Time Coupling in a Microservice Architecture
Chris Richardsonโ€ข3.8K views
Microservices Part 3 Service Mesh and Kafka by Araf Karsh Hamid
Microservices Part 3 Service Mesh and KafkaMicroservices Part 3 Service Mesh and Kafka
Microservices Part 3 Service Mesh and Kafka
Araf Karsh Hamidโ€ข7.6K views
Enterprise Architecture Toolkit Overview by Mike Walker
Enterprise Architecture Toolkit OverviewEnterprise Architecture Toolkit Overview
Enterprise Architecture Toolkit Overview
Mike Walkerโ€ข4.5K views
Microsoft DevOps Solution - DevOps by Chetan Gordhan
Microsoft DevOps Solution - DevOps Microsoft DevOps Solution - DevOps
Microsoft DevOps Solution - DevOps
Chetan Gordhanโ€ข712 views
CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom... by Bernd Ruecker
CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom...CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom...
CamundaCon 2021 Keynote :From Human Workflow to High-Throughput Process Autom...
Bernd Rueckerโ€ข396 views
Microservices Docker Kubernetes Istio Kanban DevOps SRE by Araf Karsh Hamid
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Araf Karsh Hamidโ€ข4.2K views
Cloud Operating Model Design by Joseph Schwartz
Cloud Operating Model DesignCloud Operating Model Design
Cloud Operating Model Design
Joseph Schwartzโ€ข10.1K views
TOGAF 9 Architectural Artifacts by Maganathin Veeraragaloo
TOGAF 9 Architectural ArtifactsTOGAF 9 Architectural Artifacts
TOGAF 9 Architectural Artifacts
Maganathin Veeraragalooโ€ข27.9K views
Enterprise Architecture - Overview by Faqih Zulfikar
Enterprise Architecture - OverviewEnterprise Architecture - Overview
Enterprise Architecture - Overview
Faqih Zulfikarโ€ข588 views
Event-Driven Architecture (EDA) by WSO2
Event-Driven Architecture (EDA)Event-Driven Architecture (EDA)
Event-Driven Architecture (EDA)
WSO2โ€ข10.9K views
Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro... by confluent
Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro...Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro...
Apache Kafka and ksqlDB in Action: Let's Build a Streaming Data Pipeline! (Ro...
confluentโ€ข3.7K views
Data Pipline Observability meetup by Omid Vahdaty
Data Pipline Observability meetup Data Pipline Observability meetup
Data Pipline Observability meetup
Omid Vahdatyโ€ข371 views
Ad-Tech on AWS ์„ธ๋ฏธ๋‚˜ | AWS์™€ ์‹ค์‹œ๊ฐ„ ์ž…์ฐฐ by Amazon Web Services Korea
Ad-Tech on AWS ์„ธ๋ฏธ๋‚˜ | AWS์™€ ์‹ค์‹œ๊ฐ„ ์ž…์ฐฐAd-Tech on AWS ์„ธ๋ฏธ๋‚˜ | AWS์™€ ์‹ค์‹œ๊ฐ„ ์ž…์ฐฐ
Ad-Tech on AWS ์„ธ๋ฏธ๋‚˜ | AWS์™€ ์‹ค์‹œ๊ฐ„ ์ž…์ฐฐ
Amazon Web Services Koreaโ€ข1.9K views
MLOps journey at Swisscom: AI Use Cases, Architecture and Future Vision by BATbern
MLOps journey at Swisscom: AI Use Cases, Architecture and Future VisionMLOps journey at Swisscom: AI Use Cases, Architecture and Future Vision
MLOps journey at Swisscom: AI Use Cases, Architecture and Future Vision
BATbernโ€ข220 views
Decomposing Applications for Scalability and Deployability (April 2012) by Chris Richardson
Decomposing Applications for Scalability and Deployability (April 2012)Decomposing Applications for Scalability and Deployability (April 2012)
Decomposing Applications for Scalability and Deployability (April 2012)
Chris Richardsonโ€ข18.2K views
SQL Analytics Powering Telemetry Analysis at Comcast by Databricks
SQL Analytics Powering Telemetry Analysis at ComcastSQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at Comcast
Databricksโ€ข341 views
Big Data Architectural Patterns by Amazon Web Services
Big Data Architectural PatternsBig Data Architectural Patterns
Big Data Architectural Patterns
Amazon Web Servicesโ€ข29.4K views
Application modernization patterns with apache kafka, debezium, and kubernete... by Bilgin Ibryam
Application modernization patterns with apache kafka, debezium, and kubernete...Application modernization patterns with apache kafka, debezium, and kubernete...
Application modernization patterns with apache kafka, debezium, and kubernete...
Bilgin Ibryamโ€ข748 views

Similar to Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Process Capability by
Process CapabilityProcess Capability
Process CapabilityThomas Abraham
7K viewsโ€ข41 slides
Process Mining and Predictive Process Monitoring by
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process MonitoringMarlon Dumas
3.3K viewsโ€ข40 slides
Statistical Process Control - SPC by
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPCPrasenjit Puri
153 viewsโ€ข25 slides
Six Sigma Dfss Application In Data Accarucy by
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucyxyhfun
529 viewsโ€ข28 slides
Process Mining and Predictive Process Monitoring in Apromore by
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in ApromoreMarlon Dumas
713 viewsโ€ข34 slides
A process to improve the accuracy of mk ii fp to cosmic charles symons by
A process to improve the accuracy of mk ii fp to cosmic charles symonsA process to improve the accuracy of mk ii fp to cosmic charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symonsIWSM Mensura
349 viewsโ€ข30 slides

Similar to Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models(20)

Process Capability by Thomas Abraham
Process CapabilityProcess Capability
Process Capability
Thomas Abrahamโ€ข7K views
Process Mining and Predictive Process Monitoring by Marlon Dumas
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process Monitoring
Marlon Dumasโ€ข3.3K views
Statistical Process Control - SPC by Prasenjit Puri
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPC
Prasenjit Puriโ€ข153 views
Six Sigma Dfss Application In Data Accarucy by xyhfun
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucy
xyhfunโ€ข529 views
Process Mining and Predictive Process Monitoring in Apromore by Marlon Dumas
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in Apromore
Marlon Dumasโ€ข713 views
A process to improve the accuracy of mk ii fp to cosmic charles symons by IWSM Mensura
A process to improve the accuracy of mk ii fp to cosmic charles symonsA process to improve the accuracy of mk ii fp to cosmic charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symons
IWSM Mensuraโ€ข349 views
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs by Marlon Dumas
Multi-Perspective Comparison of Business Processes Variants Based on Event LogsMulti-Perspective Comparison of Business Processes Variants Based on Event Logs
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs
Marlon Dumasโ€ข557 views
Advantages Of Linear Programming Models by Brenda Torres
Advantages Of Linear Programming ModelsAdvantages Of Linear Programming Models
Advantages Of Linear Programming Models
Brenda Torresโ€ข3 views
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op... by IRJET Journal
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
IRJET Journalโ€ข41 views
Comprehensive acquisition management methodology rev6 by clifpt
Comprehensive acquisition management methodology rev6Comprehensive acquisition management methodology rev6
Comprehensive acquisition management methodology rev6
clifptโ€ข44 views
3D-DRESD AC by Marco Santambrogio
3D-DRESD AC3D-DRESD AC
3D-DRESD AC
Marco Santambrogioโ€ข408 views
Se notes by Noor Ul Hudda Memon
Se notesSe notes
Se notes
Noor Ul Hudda Memonโ€ข1.7K views
Gestiรณn de la calidad sem 2 by youffre
Gestiรณn de la calidad sem 2Gestiรณn de la calidad sem 2
Gestiรณn de la calidad sem 2
youffreโ€ข118 views
Lpg detector final ppt by Agara Mudhalvan
Lpg detector final pptLpg detector final ppt
Lpg detector final ppt
Agara Mudhalvanโ€ข147 views
Practical Guides on Programming with Big Number Library in Scientific Researches by theijes
Practical Guides on Programming with Big Number Library in Scientific ResearchesPractical Guides on Programming with Big Number Library in Scientific Researches
Practical Guides on Programming with Big Number Library in Scientific Researches
theijesโ€ข31 views
Unbiased, Fine-Grained Description of Processes Performance from Event Data by Vadim Denisov
Unbiased, Fine-Grained Description of Processes Performance from Event DataUnbiased, Fine-Grained Description of Processes Performance from Event Data
Unbiased, Fine-Grained Description of Processes Performance from Event Data
Vadim Denisovโ€ข34 views
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING by IJCI JOURNAL
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKINGBUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
IJCI JOURNALโ€ข2 views
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN... by IAEME Publication
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
IAEME Publicationโ€ข267 views
Decisions in a supply chain modeling for comparative evaluation strategies in... by IAEME Publication
Decisions in a supply chain modeling for comparative evaluation strategies in...Decisions in a supply chain modeling for comparative evaluation strategies in...
Decisions in a supply chain modeling for comparative evaluation strategies in...
IAEME Publicationโ€ข294 views
Software Architecture in the age of Cloud Computing by Jaroslav Gergic
Software Architecture in the age of Cloud ComputingSoftware Architecture in the age of Cloud Computing
Software Architecture in the age of Cloud Computing
Jaroslav Gergicโ€ข525 views

More from Marlon Dumas

Discovery and Simulation of Business Processes with Probabilistic Resource Av... by
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Marlon Dumas
63 viewsโ€ข15 slides
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... by
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Marlon Dumas
35 viewsโ€ข19 slides
Augmented Business Process Management by
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process ManagementMarlon Dumas
1.2K viewsโ€ข23 slides
Process Mining and Data-Driven Process Simulation by
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process SimulationMarlon Dumas
299 viewsโ€ข29 slides
Modeling Extraneous Activity Delays in Business Process Simulation by
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process SimulationMarlon Dumas
148 viewsโ€ข42 slides
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints by
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsMarlon Dumas
92 viewsโ€ข23 slides

More from Marlon Dumas(20)

Discovery and Simulation of Business Processes with Probabilistic Resource Av... by Marlon Dumas
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Marlon Dumasโ€ข63 views
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... by Marlon Dumas
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Marlon Dumasโ€ข35 views
Augmented Business Process Management by Marlon Dumas
Augmented Business Process ManagementAugmented Business Process Management
Augmented Business Process Management
Marlon Dumasโ€ข1.2K views
Process Mining and Data-Driven Process Simulation by Marlon Dumas
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process Simulation
Marlon Dumasโ€ข299 views
Modeling Extraneous Activity Delays in Business Process Simulation by Marlon Dumas
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process Simulation
Marlon Dumasโ€ข148 views
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints by Marlon Dumas
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints
Marlon Dumasโ€ข92 views
Robotic Process Mining by Marlon Dumas
Robotic Process MiningRobotic Process Mining
Robotic Process Mining
Marlon Dumasโ€ข333 views
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable? by Marlon Dumas
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Marlon Dumasโ€ข51 views
Learning Accurate Business Process Simulation Models from Event Logs via Auto... by Marlon Dumas
Learning Accurate Business Process Simulation Models from Event Logs via Auto...Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Marlon Dumasโ€ข421 views
Process Mining: A Guide for Practitioners by Marlon Dumas
Process Mining: A Guide for PractitionersProcess Mining: A Guide for Practitioners
Process Mining: A Guide for Practitioners
Marlon Dumasโ€ข141 views
Optimizaciรณn de procesos basada en datos by Marlon Dumas
Optimizaciรณn de procesos basada en datosOptimizaciรณn de procesos basada en datos
Optimizaciรณn de procesos basada en datos
Marlon Dumasโ€ข386 views
Process Mining and AI for Continuous Process Improvement by Marlon Dumas
Process Mining and AI for Continuous Process ImprovementProcess Mining and AI for Continuous Process Improvement
Process Mining and AI for Continuous Process Improvement
Marlon Dumasโ€ข207 views
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction by Marlon Dumas
Prescriptive Process Monitoring for Cost-Aware Cycle Time ReductionPrescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Marlon Dumasโ€ข362 views
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio... by Marlon Dumas
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Marlon Dumasโ€ข654 views
On the Road to AI-Infused Process Execution by Marlon Dumas
On the Road to AI-Infused Process ExecutionOn the Road to AI-Infused Process Execution
On the Road to AI-Infused Process Execution
Marlon Dumasโ€ข270 views
Process Mining 2.0: From Insights to Actions by Marlon Dumas
Process Mining 2.0: From Insights to ActionsProcess Mining 2.0: From Insights to Actions
Process Mining 2.0: From Insights to Actions
Marlon Dumasโ€ข742 views
Process Mining in Action: Self-service data science for business teams by Marlon Dumas
Process Mining in Action: Self-service data science for business teamsProcess Mining in Action: Self-service data science for business teams
Process Mining in Action: Self-service data science for business teams
Marlon Dumasโ€ข407 views
Enterprise Systems by Marlon Dumas
Enterprise SystemsEnterprise Systems
Enterprise Systems
Marlon Dumasโ€ข796 views
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E... by Marlon Dumas
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Marlon Dumasโ€ข437 views
Identifying Candidate Routines for Robotic Process Automation From Unsegmente... by Marlon Dumas
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Marlon Dumasโ€ข444 views

Recently uploaded

PRIVACY AWRE PERSONAL DATA STORAGE by
PRIVACY AWRE PERSONAL DATA STORAGEPRIVACY AWRE PERSONAL DATA STORAGE
PRIVACY AWRE PERSONAL DATA STORAGEantony420421
7 viewsโ€ข56 slides
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation by
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented GenerationDataScienceConferenc1
19 viewsโ€ข29 slides
Construction Accidents & Injuries by
Construction Accidents & InjuriesConstruction Accidents & Injuries
Construction Accidents & InjuriesBisnar Chase Personal Injury Attorneys
6 viewsโ€ข5 slides
[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ... by
[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ...[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ...DataScienceConferenc1
5 viewsโ€ข19 slides
Dr. Ousmane Badiane-2023 ReSAKSS Conference by
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceDr. Ousmane Badiane-2023 ReSAKSS Conference
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceAKADEMIYA2063
5 viewsโ€ข34 slides
DGST Methodology Presentation.pdf by
DGST Methodology Presentation.pdfDGST Methodology Presentation.pdf
DGST Methodology Presentation.pdfmaddierlegum
7 viewsโ€ข9 slides

Recently uploaded(20)

PRIVACY AWRE PERSONAL DATA STORAGE by antony420421
PRIVACY AWRE PERSONAL DATA STORAGEPRIVACY AWRE PERSONAL DATA STORAGE
PRIVACY AWRE PERSONAL DATA STORAGE
antony420421โ€ข7 views
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation by DataScienceConferenc1
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
DataScienceConferenc1โ€ข19 views
Construction Accidents & Injuries by Bisnar Chase Personal Injury Attorneys
Construction Accidents & InjuriesConstruction Accidents & Injuries
Construction Accidents & Injuries
Bisnar Chase Personal Injury Attorneysโ€ข6 views
[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ... by DataScienceConferenc1
[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ...[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovatorโ€™s Dilemma: to Build or Not to ...
DataScienceConferenc1โ€ข5 views
Dr. Ousmane Badiane-2023 ReSAKSS Conference by AKADEMIYA2063
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceDr. Ousmane Badiane-2023 ReSAKSS Conference
Dr. Ousmane Badiane-2023 ReSAKSS Conference
AKADEMIYA2063โ€ข5 views
DGST Methodology Presentation.pdf by maddierlegum
DGST Methodology Presentation.pdfDGST Methodology Presentation.pdf
DGST Methodology Presentation.pdf
maddierlegumโ€ข7 views
LIVE OAK MEMORIAL PARK.pptx by ms2332always
LIVE OAK MEMORIAL PARK.pptxLIVE OAK MEMORIAL PARK.pptx
LIVE OAK MEMORIAL PARK.pptx
ms2332alwaysโ€ข7 views
4_4_WP_4_06_ND_Model.pptx by d6fmc6kwd4
4_4_WP_4_06_ND_Model.pptx4_4_WP_4_06_ND_Model.pptx
4_4_WP_4_06_ND_Model.pptx
d6fmc6kwd4โ€ข7 views
Penetration testing by Burpsuite by AyonDebnathCertified
Penetration testing by BurpsuitePenetration testing by Burpsuite
Penetration testing by Burpsuite
AyonDebnathCertifiedโ€ข5 views
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P... by DataScienceConferenc1
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
DataScienceConferenc1โ€ข8 views
CRM stick or twist workshop by info828217
CRM stick or twist workshopCRM stick or twist workshop
CRM stick or twist workshop
info828217โ€ข14 views
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo... by DataScienceConferenc1
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
DataScienceConferenc1โ€ข9 views
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx by DataScienceConferenc1
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx
DataScienceConferenc1โ€ข11 views
Data about the sector workshop by info828217
Data about the sector workshopData about the sector workshop
Data about the sector workshop
info828217โ€ข29 views
Custom Tag Manager Templates by Markus Baersch
Custom Tag Manager TemplatesCustom Tag Manager Templates
Custom Tag Manager Templates
Markus Baerschโ€ข29 views
[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines by DataScienceConferenc1
[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines
[DSC Europe 23] Luca Morena - From Psychohistory to Curious Machines
DataScienceConferenc1โ€ข5 views
Inawsidom - Data Journey by PhilipBasford
Inawsidom - Data JourneyInawsidom - Data Journey
Inawsidom - Data Journey
PhilipBasfordโ€ข8 views
Report on OSINT by AyonDebnathCertified
Report on OSINTReport on OSINT
Report on OSINT
AyonDebnathCertifiedโ€ข5 views
Lack of communication among family.pptx by ahmed164023
Lack of communication among family.pptxLack of communication among family.pptx
Lack of communication among family.pptx
ahmed164023โ€ข14 views
CRM stick or twist.pptx by info828217
CRM stick or twist.pptxCRM stick or twist.pptx
CRM stick or twist.pptx
info828217โ€ข11 views

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

  • 1. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
  • 2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
  • 3. Business Process Simulation (BPS) 3 BPS allows users to address โ€œwhat-ifโ€ analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 4. Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 5. Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 6. Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 7. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
  • 8. Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
  • 9. Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
  • 10. Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
  • 11. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
  • 12. Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
  • 13. Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
  • 14. Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
  • 15. Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
  • 16. Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
  • 17. Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth moversโ€™ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127โ€“143. Springer (2019) Process event log Simulated event log
  • 18. Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 19. Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 20. Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 21. Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 22. Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer ๐‘›, difference in the frequencies of the ๐‘›-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
  • 23. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
  • 24. Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 25. Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 26. Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am โ€“ 11am 07-10-2022 11am โ€“ 12pm
  • 27. Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
  • 28. Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am โ€“ 11am Friday 11am โ€“ 12pm
  • 29. Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
  • 30. Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
  • 31. Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 32. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
  • 33. Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am โ€“ 11am
  • 34. Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 35. Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
  • 36. Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 37. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
  • 38. Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distanceโ€™s performance significantly different from the CFLDโ€™s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
  • 39. Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 40. Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
  • 41. Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distanceโ€™s performance significantly different from the CFLDโ€™s performance? Kendall rank correlation coefficient 1.0
  • 42. Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 43. Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
  • 44. Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
  • 45. Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 46. Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 47. Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 48. Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 49. Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
  • 50. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
  • 51. Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
  • 52. Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.

Editor's Notes

  1. So, the first thing we need to know is, what is business process simulation?
  2. BPS aims to replicate the execution of a process, to mimic the behavior of the process, in a certain scenario (set of resources, etc.) analyzing its performance (KPIs) This allows usersโ€ฆ The starting point is a BPS modelโ€ฆ A process model annotated with a set of simulation parameters that define the scenario (resources, calendars, activity durationsโ€ฆ). [NEXT]
  3. BPS models may be manually created based on information collected via interviews or empirical observations.. Or [NEXTโ€ฆ]
  4. they may be automatically discovered from execution data recorded in process-aware information systems (event logs) Regardless of the origin, a key question when using a BPS model isโ€ฆ [NEXT]
  5. how to assess its quality? Several approaches have been proposed to address this problem. However, these approaches are either manual and qualitative or they produce a single number that does not allow one to identify the source(s) of deviations between the BPS model and the observed reality
  6. First we need to decide what to compare when assessing the quality of a BPS model. What we are comparing is a BPS model, with a PROCESS What we usually have isโ€ฆ
  7. โ€ฆan event log! Now, the first thing we asked ourselves was: should we compare a BPS model against a event log? But it is true that BPS models do not follow a standard structureโ€ฆ They can be formed by queue systems, but less of more models (resources, more complex waiting times), and they will change during time with new research. Thus, what we can do is simulate an event log out of the BPS model
  8. and compare log to log. K runs and compute the avg and conf int
  9. Abstract event logs into time-series or histograms and compare them
  10. We have two event logs, we are focusing on the control-flow, so the first step is toโ€ฆ [NEXT]
  11. obtain the activity sequences of each event log. Then, we compute the Damerau-Levenshtein (string edit distance) distance between each pair of casesโ€ฆ [NEXT]
  12. For exampe, [comment examples], we repeat this for each case Once we have all the pairings computed, we compute the matching between cases of one log to another (such as each case in one log is matched to one case in the other event log, with no repetitionsโ€ฆ [NEXT]
  13. While minimizing the sum of distances using the Hungarian algorithm for optimal alignment. Finally, the CFLD measure is the average of these distancesโ€ฆ [NEXT]
  14. The computational complexity of computing the DL-distance for all possible pairings is O(N2 ร—MTL3) where N is the number of traces in the logs (assuming both logs have an equal number of cases, which holds in our setting) and MTL is the maximum trace length. Since all pairings are put into a matrix to compute the optimal alignment of cases (the one that minimizes the total sum of distances), CFLDโ€™s memory complexity is quadratic on the number of cases. The optimal alignment of traces using the Hungarian algorithm has a cubic complexity on the number of cases.
  15. In the same way than for the CFLD, we are focusing on the control-flow, so the first step is to obtain the activity sequences of each event log. Leemans et al. measure the quality of a stochastic process model by mapping the model and a log to their Directly-Follows Graph (DFG), viewing each DFG as a histogram, and measuring the distance between these histograms. We note that the histogram of 2-grams of a log is equal to the histogram of its DFG. Given this observation, we generalize the approach of to n-grams, noting that the histogram of n-grams of a log is equal to the (n-1)th-Markovian abstraction of the log. Thenโ€ฆ [NEXT]
  16. Letโ€™s assume a size of N=3, so the N-grams are 3-grams (sequences of three activites). We compute all 3-grams observed in both logs, considering two dummy activities in the start and end of each trace. Thenโ€ฆ [NEXT]
  17. We measure the frequency of each N-gram in each logโ€ฆ
  18. We measure the frequency of each N-gram in each logโ€ฆ
  19. And compute the sum of absolute differences between them, normalized by the sum of frequencies of all n-grams (value between 0-1). NGD is considerably more efficient than CFLD, as the construction of the histogram of n-grams is linear on the number of events in the log, and the same goes for computing the differences between the n-gram histograms.
  20. For the temporal measures, we first do the opposite of the control-flow, we abstract from the control-flow informationโ€ฆ [NEXT]
  21. retaining only the events (in this case start and end)โ€ฆ [NEXT]
  22. Then we discretize these events into bins of 1h in the following way. Obtaining a time-series with the number of events happening in each hour of the process timeline.
  23. Once we have the temporal distribution (not a probabilistic distribution, but just the events occurring in the timeline), we compare both time-series with the EMD to measure the distance. We measure the trend.
  24. The same process is followed for the next measure, but in this case discretized to weekdays. In this way, we measure the seasonality of the events happening in the process.
  25. The temporal distribution of events of each day of the week is compared and then we compute the average distance of the 7 days.
  26. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  27. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  28. For the case arrival rate we want to measure how different the arrival of cases is. So, we retain only the events denoting the arrival of each case (start of first activity instance). Then we build the distribution in the same way than the previous metricsโ€ฆ [NEXT]
  29. and compare with EMD.