Paper presentation at the 16th International Conference on Business Process Management, Sydney, Australia, 13 September 2018. Paper available at: http://kodu.ut.ee/~dumas/pubs/bpm2018precision.pdf

1. Abstract-and-Compare:
a Family of Scalable Precision Measures
for Automated Process Discovery
Adriano Augusto, Abel Armas-Cervantes, Raffaele Conforti,
Marlon Dumas, Marcello La Rosa, and Daniel Reissner

2. Precision in Process Mining
Precision captures the extent to which the behaviour allowed by a process model
is observed in an event log:
— How much behaviour of a process model can be found in an event log?
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
2

3. State of the Art Precision Measures
Precision
Name Authors Year
Set Difference Precision Greco et Al. 2006
Advanced Behavioural Appropriateness Rozinat and van der Aalst 2008
Negative Events Precision De Weerdt et al. 2011
Alignments-based ETC precision (one-align) Adriansyah et al. 2015
Projected Conformance Checking Leemans et al. 2016
Anti-alignment Precision van Dongen et al. 2016
3

4. Five Axioms for Precision Measures
Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018).
The imprecisions of precision measures in process mining.
— Axiom 1: given a log L and a process model M,
a precision measure is a deterministic function: Prec(L, M) in ℝ.
4

5. Five Axioms for Precision Measures
Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018).
The imprecisions of precision measures in process mining.
— Axiom 1: given a log L and a process model M,
a precision measure is a deterministic function: Prec(L, M) in ℝ.
— Axiom 2: given a log L, and two process models M1 and M2.
If the behaviour of L is fully contained in M1 behaviour,
and the behaviour of M1 is fully contained in M2 behavior,
then Prec(L, M1) ≥ Prec(L, M2).
M2 M1 L Axiom 2
5

6. Five Axioms for Precision Measures
Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018).
The imprecisions of precision measures in process mining.
— Axiom 1: given a log L and a process model M,
a precision measure is a deterministic function: Prec(L, M) in ℝ.
— Axiom 2: given a log L, and two process models M1 and M2.
If the behaviour of L is fully contained in M1 behaviour,
and the behaviour of M1 is fully contained in M2 behavior,
then Prec(L, M1) ≥ Prec(L, M2).
— Axiom 3: given a log L, and two process models M1 and M2.
If the behaviour of L is fully contained in M1 and M2 is the flower process,
then Prec(L, M1) > Prec(L, M2).
M2 M1 L Axiom 2
6

7. Five Axioms for Precision Measures
Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018).
The imprecisions of precision measures in process mining.
— Axiom 1: given a log L and a process model M,
a precision measure is a deterministic function: Prec(L, M) in ℝ.
— Axiom 2: given a log L, and two process models M1 and M2.
If the behaviour of L is fully contained in M1 behaviour,
and the behaviour of M1 is fully contained in M2 behavior,
then Prec(L, M1) ≥ Prec(L, M2).
— Axiom 3: given a log L, and two process models M1 and M2.
If the behaviour of L is fully contained in M1 and M2 is the flower process,
then Prec(L, M1) > Prec(L, M2).
— Axiom 4: given a log L, and two process models M1 and M2.
If the behaviour of M1 is equal to the behaviour of M2,
then Prec(L, M1) = Prec(L, M2).
M2 M1 L Axiom 2
7

8. Five Axioms for Precision Measures
Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018).
The imprecisions of precision measures in process mining.
— Axiom 1: given a log L and a process model M,
a precision measure is a deterministic function: Prec(L, M) in ℝ.
— Axiom 2: given a log L, and two process models M1 and M2.
If the behaviour of L is fully contained in M1 behaviour,
and the behaviour of M1 is fully contained in M2 behavior,
then Prec(L, M1) ≥ Prec(L, M2).
— Axiom 3: given a log L, and two process models M1 and M2.
If the behaviour of L is fully contained in M1 and M2 is the flower process,
then Prec(L, M1) > Prec(L, M2).
— Axiom 4: given a log L, and two process models M1 and M2.
If the behaviour of M1 is equal to the behaviour of M2,
then Prec(L, M1) = Prec(L, M2).
— Axiom 5: given two logs L1 and L2, and a process model M.
If the behaviour of L1 is fully contained in L2 behaviour,
then Prec(L2, M) ≥ Prec(L1, M).
M2 M1 L
M L2 L1
Axiom 2
Axiom 5
8

9. State of the Art Precision Measures(2)
Precision Axiom Satisfied
Name Authors Year A1 A2 A3 A4 A5
Set Difference Precision Greco et Al. 2006 yes ? no yes yes
Advanced Behavioural Appropriateness Rozinat and van der Aalst 2008 no ? no yes ?
Negative Events Precision De Weerdt et al. 2011 no no ? ? ?
Alignments-based ETC precision (one-align) Adriansyah et al. 2015 no no no no no
Projected Conformance Checking Leemans et al. 2016 ? no ? ? no
Anti-alignment Precision van Dongen et al. 2016 ? ? ? ? no
Tax, N., Lu, X., Sidorova, N., Fahland, D. and van der Aalst, W. (2018).
The imprecisions of precision measures in process mining.
9

10. Why so Challenging?
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
10

11. Why so Challenging?
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
The process model behaviour
may be infinite.
The event log behaviour
is always finite.
11

12. Why so Challenging?
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
The process model behaviour
may be infinite.
The event log behaviour
is always finite.
How to fairly compare
an infinite behaviour against a finite one?
12

13. Objectives-Driven Approach Design
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
13

14. Objectives-Driven Approach Design
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
Abstract
Behaviour
Abstract
Behaviour
14

15. Objectives-Driven Approach Design
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
1. Use the same Behavioural Abstraction
2. Capture only Chunks of Behaviour
3. Control Behavioural Approximation
Abstract
Behaviour
Abstract
Behaviour
15

16. Objectives-Driven Approach Design
Event
Log
Process
Model
Event Log
Behaviour
Process
Model
Behaviour
Compare
Precision
1. Use the same Behavioural Abstraction
2. Capture only Chunks of Behaviour
3. Control Behavioural Approximation
4. Be Noise Tolerant and Rapid
5. Must satisfy the Five Axioms
Abstract
Behaviour
Abstract
Behaviour
16

17. kth-order Markovian Abstraction:
a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution.
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
1st Order Markovian Abstraction
17
Log Behaviour

18. kth-order Markovian Abstraction:
a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution.
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
1st Order Markovian Abstraction
18
Log Behaviour

19. kth-order Markovian Abstraction:
a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution.
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
1st Order Markovian Abstraction
19
Log Behaviour

20. kth-order Markovian Abstraction:
a graphical representation of behavioural chunks (i.e. subtraces) of length k and their evolution.
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
1st Order Markovian Abstraction
20
Log Behaviour

21. kth-order Markovian Abstraction:
example
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
2nd Order Markovian Abstraction
21
Log Behaviour

22. kth-order Markovian Abstraction:
example
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
2nd Order Markovian Abstraction
22
Log Behaviour

23. kth-order Markovian Abstraction:
example
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
2nd Order Markovian Abstraction
23
Log Behaviour

24. kth-order Markovian Abstraction:
example
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
2nd Order Markovian Abstraction
24
Log Behaviour

25. 2nd Order Markovian Abstraction:
from a Process Model
25

26. 26
1. Turn the process into an automaton
2nd Order Markovian Abstraction:
from a Process Model

27. 27
1. Turn the process into an automatonAutomaton
s0 s1
a
b
sf
a
b
ba
2nd Order Markovian Abstraction:
from a Process Model

28. 28
1. Turn the process into an automaton
2. Replay the automaton collecting all the
subtraces of length k
(and full traces of length < k)
Automaton
s0 s1
a
b
sf
a
b
ba
2nd Order Markovian Abstraction:
from a Process Model

29. 29
1. Turn the process into an automaton
2. Replay the automaton collecting all the
subtraces of length k
(and full traces of length < k)
Automaton
s0 s1
a
b
sf
a
b
ba
Subtraces:
<a,b> <a,a> <b,a> <b,b> <a> <b>
2nd Order Markovian Abstraction:
from a Process Model

30. 30
1. Turn the process into an automaton
2. Replay the automaton collecting all the
subtraces of length k
(and full traces of length < k)
3. Turn each subtrace into a node of the
Markovian Abstraction
Automaton
s0 s1
a
b
sf
a
b
ba
Subtraces:
<a,b> <a,a> <b,a> <b,b> <a> <b>
2nd Order Markovian Abstraction:
from a Process Model

31. 31
1. Turn the process into an automaton
2. Replay the automaton collecting all the
subtraces of length k
(and full traces of length < k)
3. Turn each subtrace into a node of the
Markovian Abstraction
Subtraces:
<a,b> <a,a> <b,a> <b,b> <a> <b>
2nd Order Markovian Abstraction
2nd Order Markovian Abstraction:
from a Process Model

32. 32
1. Turn the process into an automaton
2. Replay the automaton collecting all the
subtraces of length k
(and full traces of length < k)
3. Turn each subtrace into a node of the
Markovian Abstraction
4. Connect the nodes representing
overlapping subtraces (e.g. <a,b> and
<b,a>), and to the initial node (-) the
traces’ suffixes and prefixes.
Subtraces:
<a,b> <a,a> <b,a> <b,b> <a> <b>
2nd Order Markovian Abstraction
2nd Order Markovian Abstraction:
from a Process Model

33. 33
1. Turn the process into an automaton
2. Replay the automaton collecting all the
subtraces of length k
(and full traces of length < k)
3. Turn each subtrace into a node of the
Markovian Abstraction
4. Connect the nodes representing
overlapping subtraces (e.g. <a,b> and
<b,a>), and to the initial node (-) the
traces’ suffixes and prefixes.
Subtraces:
<a,b> <a,a> <b,a> <b,b> <a> <b>
2nd Order Markovian Abstraction
2nd Order Markovian Abstraction:
from a Process Model

34. (Graph) Comparison
— Comparator 1:
Strict Graph Comparison (edges set difference)
Precision (L, M) = 1 -
| 𝑴 𝒆 𝑳 𝒆 |
𝑴 𝒆
𝐿 𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝐿𝑜𝑔 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛
𝑀𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 𝑀𝑜𝑑𝑒𝑙 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛
34

35. (Graph) Comparison
— Comparator 1:
Strict Graph Comparison (edges set difference)
— Comparator 2 (implemented):
Hungarian Algorithm Graph Comparison (HGC), with Levenshtein Distance as cost function
Precision (L, M) = 1 -
| 𝑴 𝒆 𝑳 𝒆 |
𝑴 𝒆
𝐿 𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝐿𝑜𝑔 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛
𝑀𝑒 = 𝑒𝑑𝑔𝑒𝑠 𝑠𝑒𝑡 𝑜𝑓 𝑡ℎ𝑒 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 𝑀𝑜𝑑𝑒𝑙 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛
Precision (L, M) = 1 -
𝐻𝐺𝐶 𝑐𝑜𝑠𝑡
𝑴 𝒆
𝐻𝐺𝐶𝑐𝑜𝑠𝑡 = 𝑚𝑎𝑡𝑐ℎ𝑖𝑛𝑔 𝑐𝑜𝑠𝑡 𝑜𝑓 𝑡ℎ𝑒 𝐻𝑢𝑛𝑔𝑎𝑟𝑖𝑎𝑛 𝐴𝑙𝑔𝑜𝑟𝑖𝑡ℎ𝑚 𝑎𝑝𝑝𝑙𝑖𝑒𝑑 𝑡𝑜
𝑡ℎ𝑒 𝑀𝑎𝑟𝑘𝑜𝑣𝑖𝑎𝑛 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑃𝑟𝑜𝑐𝑒𝑠𝑠 𝑀𝑜𝑑𝑒𝑙 𝑎𝑛𝑑 𝑜𝑓 𝑡ℎ𝑒 𝐿𝑜𝑔
35
note: comparator 2 is equal to comparator 1 when the process behaviour fully contains the log behaviour..

36. kth-order Markovian Abstraction-Based Precision:
MAPk
Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
Process 1 (P1) Event Log (L) Flower Process (FP)
36

37. Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
Process 1 (P1) Event Log (L) Flower Process (FP)
MAP1 (L, P1) = 1 -
0
6
= 1.00
37
kth-order Markovian Abstraction-Based Precision:
MAPk

38. Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
Process 1 (P1) Event Log (L) Flower Process (FP)
MAP1 (L, P1) = 1 -
0
6
= 1.00
38
kth-order Markovian Abstraction-Based Precision:
MAPk

39. Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
Process 1 (P1) Event Log (L) Flower Process (FP)
MAP1 (L, P1) = 1 -
0
6
= 1.00 MAP1 (L, FP) = 1 -
2
8
= 0.75
39
kth-order Markovian Abstraction-Based Precision:
MAPk

40. Traces #
A, A, B x
A, B, B y
A, B, A, B, A, B z
MAP2 (L, P1) = 1 -
4
12
= 0.66
MAP1 (L, P1) = 1 -
0
6
= 1.00
MAP2 (L, FP) = 1 -
12
20
= 0.40
MAP1 (L, FP) = 1 -
2
8
= 0.75
40
Process 1 (P1) Event Log (L) Flower Process (FP)
kth-order Markovian Abstraction-Based Precision:
MAPk

41. Satisfiability of the Five Axioms
Precision Axiom Satisfied
Name Authors Year A1 A2 A3 A4 A5
Set Difference Precision Greco et Al. 2006 yes yes no yes yes
Advanced Behavioural Appropriateness Rozinat and van der Aalst 2008 no ? no yes ?
Negative Events Precision De Weerdt et al. 2011 no no ? ? ?
Alignments-based ETC precision (one-align) Adriansyah et al. 2015 no no no no no
Projected Conformance Checking Leemans et al. 2016 ? no ? ? no
Anti-alignment Precision van Dongen et al. 2016 ? ? ? ? no
kth-order Markovian Abstraction Augusto et al. 2018 yes yes yes* yes yes
*Axiom 3 is satisfied for a given order of k* (or higher orders).
k* = 2 for any process having at least one activity that cannot be executed twice consecutively.
41

42. Qualitative Evaluation on Artificial Data (1)
Traces #
A, B, D, E, I 1207
A, C, D, G, H, F, I 145
A, C, G, D, H, F, I 56
A, C, H, D, F, I 23
A, C, D, H, F, I 28
van Dongen, B., Carmona, J. and Chatain, T.
A unified approach for measuring precision and generalization based on anti-alignments, BPM 2016.
42
1. single trace
2. separate traces
3. flower model
4. optional G || optional H
5. G and H as self-loop activities
6. D as self-loop activity
7. all parallel activities
8. round robin

43. 43
Process
Infinite
Behaviuor
Traces
(max 2 loops)
SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7
original model no 6 7 7 9 8 7 7 7 7 = 7
single trace no 1 8 8 6 8 8 7 8 8 = 8
separate traces no 5 8 8 8 7 8 7 8 8 = 8
opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6
all parallel no 362,880 1 2 2 2 3 2 2 2 = 2
round robin yes 27 1 4 3 3 1 4 5 5 = 5
D self-loop yes 118 1 6 4 5 4 5 4 4 = 4
G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3
flower model yes 986,410 1 1 1 1 1 1 1 1 = 1
processes ordered by precision (the higher the rank the more precise the process).
Qualitative Evaluation on Artificial Data (2)

44. 44
Process
Infinite
Behaviuor
Traces
(max 2 loops)
SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7
original model no 6 7 7 9 8 7 7 7 7 = 7
single trace no 1 8 8 6 8 8 7 8 8 = 8
separate traces no 5 8 8 8 7 8 7 8 8 = 8
opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6
all parallel no 362,880 1 2 2 2 3 2 2 2 = 2
round robin yes 27 1 4 3 3 1 4 5 5 = 5
D self-loop yes 118 1 6 4 5 4 5 4 4 = 4
G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3
flower model yes 986,410 1 1 1 1 1 1 1 1 = 1
processes ordered by precision (the higher the rank the more precise the process).
Qualitative Evaluation on Artificial Data (2)

45. 45
Process
Infinite
Behaviuor
Traces
(max 2 loops)
SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7
original model no 6 7 7 9 8 7 7 7 7 = 7
single trace no 1 8 8 6 8 8 7 8 8 = 8
separate traces no 5 8 8 8 7 8 7 8 8 = 8
opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6
all parallel no 362,880 1 2 2 2 3 2 2 2 = 2
round robin yes 27 1 4 3 3 1 4 5 5 = 5
D self-loop yes 118 1 6 4 5 4 5 4 4 = 4
G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3
flower model yes 986,410 1 1 1 1 1 1 1 1 = 1
processes ordered by precision (the higher the rank the more precise the process).
Qualitative Evaluation on Artificial Data (2)

46. 46
Process
Infinite
Behaviuor
Traces
(max 2 loops)
SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7
original model no 6 7 7 9 8 7 7 7 7 = 7
single trace no 1 8 8 6 8 8 7 8 8 = 8
separate traces no 5 8 8 8 7 8 7 8 8 = 8
opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6
all parallel no 362,880 1 2 2 2 3 2 2 2 = 2
round robin yes 27 1 4 3 3 1 4 5 5 = 5
D self-loop yes 118 1 6 4 5 4 5 4 4 = 4
G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3
flower model yes 986,410 1 1 1 1 1 1 1 1 = 1
processes ordered by precision (the higher the rank the more precise the process).
Qualitative Evaluation on Artificial Data (2)

47. 47
Process
Infinite
Behaviuor
Traces
(max 2 loops)
SD ETC NE PCC AA MAP1 MAP2 MAP3 … MAP7
original model no 6 7 7 9 8 7 7 7 7 = 7
single trace no 1 8 8 6 8 8 7 8 8 = 8
separate traces no 5 8 8 8 7 8 7 8 8 = 8
opt. G || opt. H no 12 6 3 7 6 6 5 6 6 = 6
all parallel no 362,880 1 2 2 2 3 2 2 2 = 2
round robin yes 27 1 4 3 3 1 4 5 5 = 5
D self-loop yes 118 1 6 4 5 4 5 4 4 = 4
G and H self-loops yes 362 1 5 5 4 5 3 3 3 = 3
flower model yes 986,410 1 1 1 1 1 1 1 1 = 1
processes ordered by precision (the higher the rank the more precise the process).
Qualitative Evaluation on Artificial Data (2)

48. Real-Life Evaluation (setup)
48
SETUP
— 20 Real-life logs: 12 publicly available (at the 4TU data centre), and 8 proprietary
— Models discovered by three automated discovery algorithms
(Split Miner, Inductive Miner, and Structured Heuristics Miner)
— Qualitative comparison against ETC Precision (the only feasible in a real-life context)
— Time performance comparison against ETC Precision

49. Real-Life Evaluation (results)
49
RESULTS
— MAPk easily distinguishes between process models with poor precision and high precision
— MAPk is suitable for quality assessment and (especially) comparison of process models
— MAPk can be over 10 times faster than ETC
(avg. time 3.7s vs 60.0s, on models discovered by Split Miner)

50. Inductive Miner – SEPSIS Log
Split Miner – SEPSIS Log
50
Real-Life Evaluation
qualitative example

51. Inductive Miner – SEPSIS Log
Split Miner – SEPSIS Log
51
Real-Life Evaluation
qualitative example

52. Real-Life Evaluation
qualitative example
Inductive Miner – SEPSIS Log
Split Miner – SEPSIS Log
52
Split Miner Inductive Miner
ETC 0.859 0.445
MAP2 1.000 0.226
MAP3 1.000 0.051
MAP4 1.000 0.009

53. Limitations
53
—The order k and the execution time are proportional
—How to choose k?
—The selection of the best k is not automated

54. Future Work
—Designing a complementary Markovian Abstraction-Based Fitness
—Exploring alternative comparison algorithms (e.g. graph bisimulation)
—Using the Markovian Precision in automated process discovery
for reinforcement learning
54

55. Thanks for Attending!
Questions?
55