Slides of my presentation at BPMS2 workshop at BPM conference, 10 September 2018, Sydney, Australia

1. 1/15 | www.janclaes.info
Jeroen Bolle and Jan Claes
Investigating the trade-off between the
effectiveness and efficiency of process
modeling

2. 2/15 | www.janclaes.info
Motivation
Previous research
 Differentiated process modeling technique
 Proposed effects on effectiveness and efficiency
Dataset and info can be downloaded at http://www.janclaes.info/experiments
February 2015
Ghent University
146 master students
Business Engineering
BPM course

3. 3/15 | www.janclaes.info
Motivation
Jan Claes, Irene Vanderfeesten, Frederik Gailly, Paul Grefen, Geert Poels, The Structured Process Modeling Method
(SPMM) - What is the best way for me to construct a process model?, Decision Support Systems, Vol 100, p. 57-76, 2017
EFFECTIVENESS EFFICIENCY
Low R2 values High R2 values

4. 4/15 | www.janclaes.info
Research question
Is there a trade-off between effectiveness and
efficiency in process modeling?
 Is there a revision phase at the end of modeling?
(revising = investing time and effort = lower efficiency)
 Does this revision phase result in higher quality?
(revising = correcting errors = higher effectiveness)

5. 5/15 | www.janclaes.info
Measurements
 Time: number of seconds in timeframe
 Effort: number of operations in timeframe
 Syntactic quality
 Number of syntax errors made in timeframe
 Number of syntax errors solved in timeframe
Dataset of this paper at DOI 10.17632/5b8by4k244.1
For syntax error determination in partial models, see De Bock & Claes (2018)

6. 6/15 | www.janclaes.info
Results
Numberoferrorsper100elementsinfinalmodel
Total modeling time
Errors made
Errors solved
Errors remaining

7. 7/15 | www.janclaes.info
Results
Total time Total effort
Errors made -0,200
(p=0,820)
-0,003
(p=0,970)
Errors solved 0,253*
(p=0,050)
0,349**
(p=0,000)
Errors remaining -0,183*
(p=0,044)
-0,229*
(p=0,012)

8. 8/15 | www.janclaes.info
Revision phase
How to determine time ‘t’ ?
 Modeling phase: most of the creation
 Revision phase: moving, deleting, or renaming
elements, extensive scrolling
 Revision phase: also creates after deletes or
to close gaps etc.
Result: 81 of 121 modelers had a revision phase
Modeling phase Revision phase
t

9. 9/15 | www.janclaes.info
Revision phase
RevisionModeling RevisionModeling

10. 10/15 | www.janclaes.info
Results
Errors made in revision phase
minus
errors made in modeling phase
Errors solved in revision phase
minus
errors solved in modeling phase
Relatively more
errors made in
modeling phase
per100elementsperseconds

11. 11/15 | www.janclaes.info
Results
Errors made in revision phase
minus
errors solved in revision phase
Errors made in revision phase
and
Errors solved in revision phase
21%
Relatively more
errors made
than solved in
revision phase
per100elementsperseconds

12. 12/15 | www.janclaes.info
Conclusions
Is there a trade-off between effectiveness and
efficiency in process modeling?
 No relation between time/effort and errors made
 Relation between time/effort and errors solved
Is there a revision phase at the end of modeling?
 81 of 121 modelers (66%) had a revision phase
Does this revision phase result in higher quality?
 48% higher, 41% equal, 21% lower number of
errors

13. 13/15 | www.janclaes.info
Discussion
Only mild support for hypotheses
 33% no revision phase
 14% contra-productive revision phase
 28% no productive revision phase
 25% productive revision phase

14. 14/15 | www.janclaes.info
Limitations
 Difficult comparison
 We normalized to model size and model duration
 We did not normalize to number of errors made
 No discrimination between error types
 Solving a major error while introducing a minor error…
 Only syntax errors
 Most objective to measure, but least relevant?
 Subjective measures by limited number of people
 E.g., naïve revision phase approach
 Limited external validity
 121 master students Business Engineering at UGent

15. 15/15 | www.janclaes.info
 Do you have any questions?
 Do you have feedback?
Thanks for you attention!
Jan Claes
jan.claes@ugent.be
www.janclaes.info