Conformance checking between designed and observed processes

Conformance
Checking Between
Designed and
Observed Processes
(My Viennese Christmas Sketches)
Artem Polyvyanyy
polyvyanyy.com | @uartem
1

Process Mining
Artem Polyvyanyy | Vienna, Austria | December 2018 2
Wil M. P. van der Aalst et al. Process Mining Manifesto. Business Process Management Workshops (1) 2011: 169-194

Quality Measures in Process Mining
3
- fitness
+ precision
- generalization
+ simplicity
+ fitness
+ precision
- generalization
+/- simplicity
+ fitness
- precision
+/- generalization
+/- simplicity
+ fitness
+ precision
+ generalization
+ simplicity
Adapted from Matthias Weidlich

Quality Measures in Process Mining
4
- fitness
+ precision
- generalization
+ simplicity
+ fitness
+ precision
- generalization
+/- simplicity
+ fitness
- precision
+/- generalization
+/- simplicity
+ fitness
+ precision
+ generalization
+ simplicity
Adapted from Matthias Weidlich

5
Artem Polyvyanyy
Precision and Recall
for Process Mining
(under review)
Artem Polyvyanyy,
Andreas Solti,
Matthias Weidlich,
Claudio Di Ciccio, and
Jan Mendling

Precision and Recall
Precision: Fraction of the shared
behaviour relative to all
the behaviour of the model
Recall (a.k.a Fitness):
Fraction of the shared
behaviour relative to all
the behaviour of the log
Log Model
Log ∩ Model
m(Log ∩ Model)
m(Model)
m(Log ∩ Model)
m(Log)
How to define measure m and
operation ∩ ?
Which properties should
precision and recall have?

Imprecise Precisions
Niek Tax, Xixi Lu, Natalia Sidorova, Dirk Fahland, Wil M. P. van der Aalst:
The imprecisions of precision measures in process mining. Inf. Process. Lett. 135: 1-8 (2018)
A1: Precision is deterministic
A3: Precision between a log and a
flower model is the lowest
A4: Precision of a log on two language
equivalent models should be equal
A2:
A5:
m(L1 ∩ L3)
m(L3)
m(L2 ∩ L3)
m(L3)
≤
m(L1 ∩ L3)
m(L3)
m(L1 ∩ L2)
m(L2)
≤

Entropy-Based Precision and Recall
 The work co-authored by A. Polyvyanyy, A. Solti, M. Weidlich, C. Di Ciccio, and J. Mendling
 The only (to the best of my knowledge) precision measure that satisfies all the
proposed by the process mining community properties
 Under review, pre-pint available at: https://arxiv.org/pdf/1812.07334.pdf
 Based on the notion of topological entropy of an irreducible regular language, a language of
an ergodic, i.e., strongly connected, deterministic finite automaton (DFA)
 Let Cn(B) be the set of all words of length n in the language L(B) of a DFA B:
 Topological entropy can be computed as a Perron-Frobenius eigenvalue, i.e., a largest
eigenvalue, of the adjacency matrix of B; also if B describes an infinite language
 Logs and models get encoded as DFAs
 Model and log DFAs are short-circuited to obtain ergodic automata
 Overlap of DFAs is a well-defined operation and is used to compute the language in the
intersection of the model and log
 Minimization of DFAs is not required but empirically shown to speed up the computation of
eigenvalues

Precise Entropy-Based Precision
A1: Precision is deterministic
A3: Precision between a log and a
flower model is the lowest
A4: Precision of a log on two language
equivalent models should be equal
A2:
A5:
ent (L1 ∩ L3)
ent(L3)
ent(L2 ∩ L3)
ent(L3)
<
ent (L1 ∩ L3)
ent(L3)
ent(L1 ∩ L2)
ent(L2)
<
Entropy-based precision
note the strict monotonicity;
also fulfils additional properties
(to be shown in a few slides)

a{<min>,<max>} is short-hand for enumerating the
minimal and maximal number of repetitions of a, e.g.,
a{0,2}○b encodes language [<b>,<a,b>,<a,a,b>].

fits model

ATAED’18
recent precent
✓ ✓
✓ ✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Conjecture
Undefined for
infinite languages
Difficult
case
A1
A4
~A5

Yet Another Precision Measure
 Uses the k-th order Markovian abstraction
 Claims fulfilment of the 5 “weak” axioms by Tax et al.
(i.e., A2 and A5 are based on ≤)
 Parameter k for which axioms hold is specific for a
given set of models and logs
 Gets computationally demanding as k gets large
 A k-th order Markovian abstraction can be seen as a
behavioural profile
 Behavioral profiles are less expressive than regular
languages, refer to A.Polyvyanyy et al.: On the
expressive power of behavioral profiles. Formal Asp.
Comput. 28(4): 597-613 (2016)
 Measures for comparing behaviours based on beha-
vioural profiles were first proposed in M.Weidlich et al.:
Process compliance analysis based on behavioural
profiles. Inf. Syst. 36(7):1009-1025 (2011)
M1-abstraction M2-abstraction
BPM’18

Entropy-Based Recall
 Given a sequential model of 10 activities and a ﬁtting event log with no noise, we
gradually increase the amount of noisy traces in the event log
 Noise is deﬁned as removing, adding, or swapping events in the event log

Implementation and Evaluation
 Event logs are available via: https://data.4tu.nl/repository/collection:event_logs_real
 Eigenvalue computation implementation available via:
https://github.com/andreas-solti/matrix-toolkits-java
 Experiments and implementation available via:
https://github.com/andreas-solti/eigen-measure
Source code:
Scalability evaluation setup:
 For each log, mined a model with InductiveMiner (noise threshold 0.2)
 In all but two of the real-life cases our method computes the automata and their largest
eigenvalues within 10 minutes to yield the precision and recall values

Conclusion and Future Work
 The only (to the best of my knowledge) precision measure that satisfies all
the proposed by the process mining community properties
 Under review, pre-pint available at: https://arxiv.org/pdf/1812.07334.pdf
Possible avenues for future work include:
 Extend the measures to support partial trace matching (in progress)
 Extend the measures to support infinite-state behaviors (initial idea available)
 Improve efficiency (use efficient implementations/optimizations)
 Extend the measures to support frequencies of traces (under review)
 Applications of the measure in Process Mining and other areas,
e.g. Software Engineering
Limitations:
 Computation efficiency can be improved (though already feasible)
 Partial trace matching is not supported (desired properties of such measures
are yet to be understood)
 Infinite-state behaviors are not supported
 Frequencies of traces are not accounted for (desired properties of such
measures are yet to be understood)

Other Interests
 Process Querying including languages for process querying.
 Series of workshops: http://processquerying.com/
 Process Query Language (PQL): https://github.com/processquerying/PQL
 Process Querying Framework: Artem Polyvyanyy, Chun Ouyang, Alistair Barros,
Wil M. P. van der Aalst: Process querying: Enabling business intelligence through
query-based process analytics. Decision Support Systems 100: 41-56 (2017)
 Edited book project
 Process Forecasting: Rouven Poll, Artem Polyvyanyy, Michael Rosemann, Maximilian
Röglinger, Lea Rupprecht: Process Forecasting: Towards Proactive Business Process
Management. BPM 2018: 496-512
 Stochastic Process Mining: Work on stochastic precision and recall with Sander Leemans
 Information Systems Modeling: Work on a formalism for encoding processes and data
with Jan Martijn van der Werf:
 Language for modeling information systems for which reachability is decidable
http://www.cs.uu.nl/research/techreps/repo/CS-2018/2018-004.pdf
 Isotactics: Artem Polyvyanyy, Jan Sürmeli, Matthias Weidlich: Interleaving isotactics - An
equivalence notion on behaviour abstractions. Theor. Comput. Sci. 737: 1-18 (2018)

Process Querying Framework

Edited book entitled
“Process Querying Methods”
 To appear in mid-2019!
 53 contributors: Ahmed Awad, Alexander Artikis, Amal Elgammal, Amal Fawzy
Elgammal, Amin Beheshti, Andreas Oberweis, Andreas Schoknecht, Antonia M.
Reina, Antonio Cancela, Artem Polyvyanyy, Boualem Benatallah, Carl Corea, Chiara Di
Francescomarino, Christoph Drodt, David Knuplesch, Dennis Riehle, Eduardo
González López de Murillas, Emiliano Reynares, Fabrizio Smith, Farhad Amouzgar,
Francesco Taglino, Hajo A. Reijers, Hamid R. Motahari Nezhad, Han van der Aa,
Harald Störrle, Jianmin Wang, Jorge Roa, Jose Miguel Pérez, Kazimierz Subieta, Klaus
Kammerer, Luisa Parody, Manfred Reichert, María Laura Caliusco, María Teresa
Gómez, Mariusz Momotko, Matthias Weidlich, Maurizio Proietti, Oktay Turetken,
Pablo Villarreal, Paolo Tonella, Patrick Delfmann, Peter Fettke, Ralf Laue, Remco
Dijkman, Rik Eshuis, Rüdiger Pryss, Samira Ghodratnama, Stefanie Rinderle-Ma,
Steffen Höhenberger, Tao Jin, Tom Thaler, Vlad Acretoaie, and Wil M. P. van der Aalst.
21

Isotactics

23
Impact-Driven
Process Model
Repair
Artem Polyvyanyy,
Wil M. P. van der Aalst,
Arthur H. M. ter Hofstede, and
Moe T. Wynn
Artem Polyvyanyy

Design vs Experience

Conformance Checking
a
p1
t1 p2
b c
d e
p3
p4
p5
t2 t3
t4 t5
a b c
a b c
t1 t2 t3
γ1 =
a d e
a d e
t1 t4 t5
γ2 =
perfect alignments
a b x
a b
t1 t2
γ3 =
real-world traces
designed traces
e ≫
c
t3
≫ ≫
a b x
a b
t1 t2
γ4 =
e≫
c
t3
≫ ≫
a b
a
t1 t4
γ5 = ≫d
≫ x e
e
t5
≫
0 0 1 1 2 0 0 1 2 1 0 3 1 1 0+ + + + + + + + == =4 4 5+ + + +
move on trace move on modelsynchronous move
cost(γ3)=cost(γ4)=4 and cost(γ5)=5; γ3 and γ4 are optimal alignments!

Process Model Repair
a
p1
t1 p2
b c
d e
p3
p4
p5
t2 t3
t4 t5
a b x
a b
t1 t2
γ4 =
e≫
c
t3
≫ ≫
0 0 1 2 1
γ6 =
a e
a e
t1 t5
≫
d
t4
0 3 0
 = 7R=({x},{d})
insert skip
x t6
t7
γ7 =
a
a
t1
0
b
b
t2
0
x
x
t6
0
c
t3
2
≫
1
e
≫ γ9 =
a
a
t1
0
≫
t7
0
b
≫
1
≫
1
x
0
e
e
t5
γ8 =
a
a
t1
0
≫
t7
0 0
e
e
t5
+ + + + = 3 + + + + = 2 + + = 0
 = 2
Note the problem with x!
ττ
γ8 and γ9 are optimal alignments!

Towards Optimal Repair
R=({x},{d})
a
p1
t1 p2
b c
d e
p3
p4
p5
t2 t3
t4 t5
γ10 =
a
a
t1
0
d
t4
0
≫
1
b
≫
0
x
≫
e
e
t5
0
 = 1
=
a e
a e
t1 t5
≫
d
t4
0 0 0
γ11
simulating repair via “free” moves
a
p1
t1 p2
b c
d e
p3
p4
p5
t2 t3
t4 t5
t7
x t6
 = 1
γ12 =
a
a
t1
0
≫
t7
0
b
≫
1 0
e
e
t5
+ + + + = 1
x
x
t6
0
γ8 =
a
a
t1
0
≫
t7
0 0
e
e
t5
+ + = 0
ττ
Note no problem with x!

Optimal Repair (Theorem 5.3)
There exist optimal alignments between
the traces and the repaired model which:
(i) Demonstrate that the traces fit the
repaired model “at least as good” as they
fit the original model;
(ii) “Fulfil” the repair recommendation,
e.g., γ8 and γ12 contain no “bad” moves
(x,≫) and (≫,d).
a
p1
t1 p2
b c
d e
p3
p4
p5
t2 t3
t4 t5
a
p1
t1 p2
b c
d e
p3
p4
p5
t2 t3
t4 t5
t7
x t6
 = 1
γ12 =
a
a
t1
0
≫
t7
0
b
≫
1 0
e
e
t5
+ + + + = 1
x
x
t6
0
γ8 =
a
a
t1
0
≫
t7
0 0
e
e
t5
+ + = 0
ττ
R=({x},{d})
Note no problem with x!

One More Example
a
p1
c
d
b f
g
h
e
t1
t2
t3
t4
t5
t6
t7
t8
t9
t10
t11
p2
p3
p4 p5
p6
p7
p8
p9
p10 p11
10 x
9 x
6 x
2 x
9 x
7 x
2 x
 = 120
All deviations
cost one Peso!

A Possible Repair
R = ({f,e,x},{c,f,g})
 = 47
insert skip
a
p1
c
d
b
f
g
he
t1
t2
t3
t4
t5
t6
t7
t8
t9
t10
t11
p2 p3
p4 p5 p6
p7
p8
p9
p10 p11
t12
t13
t14
x
f
e
t15
t16 t17

Another Possible Repair
R = ({f,e},{g,d,e,c})
 = 33
insert skip
a
p1
c
d
b
f
g
h
e
t1
t2
t3
t4
t5
t6
t7
t8
t9
t10
t11
p2
p3
p4
p5
p6
p7
p8
p9
p10
p11
t13
t15
t16
t17
t18
e f
t12 t13
et14

Towards Optimal Repair Recommendation
R = ({f,x},{d,e,c,h})
 = 25
insert skip
a
p1
c
d
b
f
g
he
t1
t2
t3
t4
t5
t6
t7
t8
t9
t10
t11
p2
p3
p4
p5
p6
p7
p8
p9
p10
p11
t13
t12
f
x
t14
f
t15
t16
t17
t18
costs of repair
recommendation
implemented
heuristics
numbers of discovered
recommendations
numbers of alignment
computations deviation costs

Conclusion and Future Work
In this work, we …
 Defined, and proposed a solution to, the optimal process model repair problem
 Defined, and proposed several (approximate) solutions to, the optimal repair
recommendation problem
 Developed a publicly available tool that implements all the invented techniques,
https://svn.win.tue.nl/repos/prom/Packages/SelectivePRepair/Trunk/release/
 Reported on lessons learned from experiments with real-world data, e.g.,
for small repairs proposed heuristics perform just as well as brute-force
Possible avenues for future work include:
 Case studies to better understand requirements of process model repair
 Aesthetic process model repair
 Repair heuristics with guarantees
 Repairs based on other conformance data structures
 Repairs that go beyond trace fitness
 Applications of the proposed repair technique

Conformance checking between designed and observed processes

More Related Content

Similar to Conformance checking between designed and observed processes

Recently uploaded

Conformance checking between designed and observed processes