This slide tells you how to measure precision between process model and its recorded execution. Presented at BPI 2012, Tallinn, Estonia, by Arya Adriansyah. The technique is implemented in ProM 6.x, package ETConformance (see http://processmining.org).

1. Tallinn, 3 September 2012
Alignment-based
Precision Checking
A. Adriansyah1, J. Munoz Gamma2,
J. Carmona2, B.F. van Dongen1,
W.M.P. van der Aalst1
1) Department of Mathematics and Computer Science,
Eindhoven University of Technology, The Netherlands
2) Software Department, Universitat Politècnica de
Catalunya, Spain

2. Model needs to be precise
Event Log
aabd
abd
c
acd
a d
aabd
a b a b
a b d end
end
c d c
Process Models
1

3. Unfitting
Event Log a 0 …
Prefix automaton
b d
2 2 2
a
c
aabd 0 …
…
abd a b
1
d
1
4 4
c
acd c
How to
a
aabd
d d measure precision cfor unfitting
b … 0
0 d
1 1
event logs? c
0 d
d …
0
a b 0
a b
a b d end
end
Reference: J. Muñoz-Gama, J. Carmona (2010). A
c d
Fresh Look at Precision in Process Conformance.
8th International Conference of Business Process
c
Process Models
Management (BPM): 211-226.
2

4. Overview
Unfitting Log
aabbd
adabd
ad Alignment Fitting Traces Alignment Automaton
…
aabd
aabd
acd
…
Process Model
Experiments Precision
Metrics
3

5. Optimal alignment: least #deviations
Trace adab aabd Reference: W.M.P. van der Aalst, A.
Adriansyah, B.F. van Dongen (2012).
Replaying history on process models
for conformance checking and
performance analysis. WIREs Data
Mining and Knowledge
Alignment
ad a b Discovery, 2(2), 182-192.
a a b d
Process Model a b
a b d
end
c
4

6. Optimal alignments can be many
Trace ad abd or acd
Alignment 1
a d Alignment 2
a d
ab d ac d
Both alignments are optimal
Process Model a b
a b d
end
c
5

7. Overview
Unfitting Log
aabbd
adabd
ad Alignment Fitting Traces Alignment Automaton
…
aabd
aabd
acd
…
Process Model
Experiments Precision
Metrics
6

8. All-Alignment Automaton (Prefix-based)
Fitting Traces All-Alignments Automaton
aabd b d
a 2
1 1
2 1
2
abd
a b
abd or acd 2
43 3
4
2 1 d 1.5
1.5 1
aabd
c d
0.5 0.5
a b
a b d
end
c
Process Model
7

9. 1-Alignment automaton (Prefix-based)
Fitting Traces 1-Align Automaton
aabd b d
a 2 2 2
abd
abd or acd a b d
4 4 2 2
aabd
a b
a b d
end
c
Process Model
8

10. 1-Alignment automaton (Prefix-based)
Fitting Traces 1-Align Automaton
aabd b d
a 2 2 2
abd
abd a b d
4 4 2 2
aabd
c
0
Imprecision
a b
a b d
end
c
Process Model
9

11. Comparison of log automata
Without alignment
Event Log Fitting Traces
Prefix-based Log Automata
aabd aabd
abd abd
ad abd/acd All-Alignments
adab aabd
a b
a b d end 1-Alignment
c
Process Model
10

12. Overview
Unfitting Log
aabbd
adabd
ad Alignment Fitting Traces Alignment Automaton
…
aabd
aabd
acd
…
Process Model
Experiments Precision
Metrics
Reference: J. Muñoz-Gama, J. Carmona (2010). A Fresh Look at Precision in
Process Conformance. 8th International Conference of Business Process
Management (BPM): 211-226.
11

13. Overview
Unfitting Log
aabbd
adabd
ad Alignment Fitting Traces Alignment Automaton
…
aabd
aabd
acd
…
Process Model
Experiments Precision
Metrics
12

14. Implementation in ProM 6
• Package: “ETConformance”
13

15. Experiment settings
Event Log Duplicate Log Combined Log
Flower (F) Flower’ (F’)
Precise-Precise (PP’)
Precise-Flower
(PF’)
Precise (P) Precise’ (P’) Flower -Precise
(FP’)
Flower-Flower
(FF’)
14

16. Experiment 1: Measuring precision stability
Perfectly fitting logs and models Non-fitting logs and models
P FP’ PF’ PP’ P FP’ PF’ PP’
15

17. Experiment 2: Sensitivity to unfitting trace
Model P Model PF’
1.00 1.00
0.80
ETC
0.95
Precision
Precision
0.60
0.90 1-Align
0.40
0.85 0.20
All-Align
0.80 0.00
0 1 2 3 4 0 1 2 3 4
#Removed events #Removed events
0 1
Model FP’ 2 3 4
Model PP’
0.40 1.00
Number of Removed Events •
Precision
0.30 0.98
Precision
0.20 0.95
0.10 0.93
0.00 0.90
0 1 2 3 4 0 1 2 3 4
#Removed events #Removed events
16

18. Experiment 3: Real life cases
•
17

19. Overview
Unfitting Log
aabbd
adabd
ad Alignment Fitting Traces Alignment Automaton
…
aabd
aabd
acd
…
Process Model
Experiments Precision
Metrics
18

20. Conclusion and Future Work
•
19

21. Questions
20

22. Thanks!
21