This talk presents the design and evaluation of the BPMN Sketch Miner, a tool which uses a textual DSL as an input and by combining mining with textual modelling produces a visual BPMN process model at real time as the user is typing the input. https://www.bpmn-sketch-miner.ai/ Paper presented at the 23rd International Conference on Model Driven Engineering Languages and Systems (MODELS), Montreal, Canada, October 2020 available at http://design.inf.usi.ch/publications/2020/models.

1. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
From Text to Visual BPMN Process Models:
Design and Evaluation
Ana Ivanchikj, Souhaila Serbout and Cesare Pautasso
Software Institute,USI
Lugano, Switzerland
October, 2020
BPMN Sketch Miner October, 2020 1 / 59

2. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Classrooms
Theory Lecture
BPMN Sketch Miner October, 2020 2 / 59

3. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Classrooms
Theory Lecture Practice Lecture
BPMN Sketch Miner October, 2020 3 / 59

4. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Classrooms
Theory Lecture Practice Lecture
Challenges
Memorise the semantics of BPMN constructs
BPMN Sketch Miner October, 2020 4 / 59

5. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Classrooms
Theory Lecture Practice Lecture
Challenges
Memorise the semantics of BPMN constructs
Abstract from process instance to process model
BPMN Sketch Miner October, 2020 5 / 59

6. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Companies
Analysis and Modelling
Business Analyst
BPMN Sketch Miner October, 2020 6 / 59

7. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Companies
Interviews
Stakeholders and Business Analysts
Analysis and Modelling
Business Analyst
BPMN Sketch Miner October, 2020 7 / 59

8. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Companies
Interviews
Stakeholders and Business Analysts
Analysis and Modelling
Business Analyst
Challenges :
Receive immediate feedback on the designed process model
BPMN Sketch Miner October, 2020 8 / 59

9. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Companies
Interviews
Stakeholders and Business Analysts
Analysis and Modelling
Business Analyst
Challenges :
Receive immediate feedback on the designed process model
Reduce the cognitive load of the BA during the interviews
BPMN Sketch Miner October, 2020 9 / 59

10. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling AS-IS Situation
BPMN Modelling Today - in Companies
Interviews
Stakeholders and Business Analysts
Analysis and Modelling
Business Analyst
Challenges :
Receive immediate feedback on the designed process model
Reduce the cognitive load of the BA during the interviews
Adjust graph layout when need for changes in the process model is identified
BPMN Sketch Miner October, 2020 10 / 59

11. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling Alternative Approach
Textual Modelling with Mining
BPMN Sketch Miner October, 2020 11 / 59

12. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling Alternative Approach
Textual Modelling with Mining
BPMN Sketch Miner October, 2020 12 / 59

13. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling Alternative Approach
Textual Modelling with Mining
BPMN Sketch Miner October, 2020 13 / 59

14. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling Alternative Approach
Textual Modelling with Mining
BPMN Sketch Miner October, 2020 14 / 59

15. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Modelling Alternative Approach
Textual Modelling with Mining
BPMN Sketch Miner October, 2020 15 / 59

16. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Design Decisions
Design Decisions
BPMN Sketch Miner October, 2020 16 / 59

17. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Design Decisions
Design Decisions
BPMN Sketch Miner October, 2020 17 / 59

18. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Design Decisions
Design Decisions
BPMN Sketch Miner October, 2020 18 / 59

19. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Design Decisions
Design Decisions
BPMN Sketch Miner October, 2020 19 / 59

20. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Design Decisions
Design Decisions
BPMN Sketch Miner October, 2020 20 / 59

21. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Design Decisions
Design Decisions
Challenges
Trade-off between expressiveness and learnability
Right balance of mining vs modelling
BPMN Sketch Miner October, 2020 21 / 59

22. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Constructs Support
BPMN Constructs Support
Challenges
Trade-off between expressiveness and learnability
Right balance of mining vs modelling
Missing Constructs
Subprocesses / call activities / activity markers
Some gateways (inclusive, complex, parallel event based)
Some event types (error, compensation, conditional)
Boundary events
Data objects and data flow
BPMN Construct
Derived
from mining
Explicitly
modeled
DSL construct
tasks state each task name in a new line
task type use the “manual”, “user”, “script”, “rule”, “service”, “send”, “receive” keywords
events
use “( )” to indicate an event. Whether it is a start, intermediate or end event
is automatically inferred.
event type
use the “time”, “error”, “receive”, “send”, “notify”, “publish”, “escalate”,
“terminate” keywords
link events derived from the lack of a matching fragment
exclusive gateway split/merge repeat the last common task before the split
labeled exclusive gateway split use “?” after the name of the label
loops repeat the tasks in the loop
sequence flow state tasks in the order in which they happen
conditional flow state the condition following the gateway label
message flow use matching names for the throw and catch events
event-based gateway the last common task before the split has to be followed by events
parallel AND gateway separate the parallel tasks with “|”
lanes state the name of the lane followed by “ :” and a task
pools lanes become pools in case of message exchange
text annotation use “//” before the task to which the text annotation is to be attached
BPMN Sketch Miner October, 2020 22 / 59

23. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Constructs Support
BPMN Constructs Support
Challenges
Trade-off between expressiveness and learnability
Right balance of mining vs modelling
BPMN Construct
Derived
from mining
Explicitly
modeled
DSL construct
tasks state each task name in a new line
task type use the “manual”, “user”, “script”, “rule”, “service”, “send”, “receive” keywords
events
use “( )” to indicate an event. Whether it is a start, intermediate or end event
is automatically inferred.
event type
use the “time”, “error”, “receive”, “send”, “notify”, “publish”, “escalate”,
“terminate” keywords
link events derived from the lack of a matching fragment
exclusive gateway split/merge repeat the last common task before the split
labeled exclusive gateway split use “?” after the name of the label
loops repeat the tasks in the loop
sequence flow state tasks in the order in which they happen
conditional flow state the condition following the gateway label
message flow use matching names for the throw and catch events
event-based gateway the last common task before the split has to be followed by events
parallel AND gateway separate the parallel tasks with “|”
lanes state the name of the lane followed by “ :” and a task
pools lanes become pools in case of message exchange
text annotation use “//” before the task to which the text annotation is to be attached
BPMN Sketch Miner October, 2020 23 / 59

24. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Constructs Support
BPMN Constructs Support
Challenges
Trade-off between expressiveness and learnability
Right balance of mining vs modelling
Explicitly modelled
First-class citizen constructs (tasks and lanes)
BPMN Construct
Derived
from mining
Explicitly
modeled
DSL construct
tasks state each task name in a new line
task type use the “manual”, “user”, “script”, “rule”, “service”, “send”, “receive” keywords
events
use “( )” to indicate an event. Whether it is a start, intermediate or end event
is automatically inferred.
event type
use the “time”, “error”, “receive”, “send”, “notify”, “publish”, “escalate”,
“terminate” keywords
link events derived from the lack of a matching fragment
exclusive gateway split/merge repeat the last common task before the split
labeled exclusive gateway split use “?” after the name of the label
loops repeat the tasks in the loop
sequence flow state tasks in the order in which they happen
conditional flow state the condition following the gateway label
message flow use matching names for the throw and catch events
event-based gateway the last common task before the split has to be followed by events
parallel AND gateway separate the parallel tasks with “|”
lanes state the name of the lane followed by “ :” and a task
pools lanes become pools in case of message exchange
text annotation use “//” before the task to which the text annotation is to be attached
BPMN Sketch Miner October, 2020 24 / 59

25. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Constructs Support
BPMN Constructs Support
Challenges
Trade-off between expressiveness and learnability
Right balance of mining vs modelling
Explicitly modelled
First-class citizen constructs (tasks and lanes)
BPMN Construct
Derived
from mining
Explicitly
modeled
DSL construct
tasks state each task name in a new line
task type use the “manual”, “user”, “script”, “rule”, “service”, “send”, “receive” keywords
events
use “( )” to indicate an event. Whether it is a start, intermediate or end event
is automatically inferred.
event type
use the “time”, “error”, “receive”, “send”, “notify”, “publish”, “escalate”,
“terminate” keywords
link events derived from the lack of a matching fragment
exclusive gateway split/merge repeat the last common task before the split
labeled exclusive gateway split use “?” after the name of the label
loops repeat the tasks in the loop
sequence flow state tasks in the order in which they happen
conditional flow state the condition following the gateway label
message flow use matching names for the throw and catch events
event-based gateway the last common task before the split has to be followed by events
parallel AND gateway separate the parallel tasks with “|”
lanes state the name of the lane followed by “ :” and a task
pools lanes become pools in case of message exchange
text annotation use “//” before the task to which the text annotation is to be attached
BPMN Sketch Miner October, 2020 25 / 59

26. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Constructs Support
BPMN Constructs Support
Challenges
Trade-off between expressiveness and learnability
Right balance of mining vs modelling
Explicitly modelled
First-class citizen constructs (tasks and lanes)
Parallelism (stating all possible permutations is too time consuming)
BPMN Construct
Derived
from mining
Explicitly
modeled
DSL construct
tasks state each task name in a new line
task type use the “manual”, “user”, “script”, “rule”, “service”, “send”, “receive” keywords
events
use “( )” to indicate an event. Whether it is a start, intermediate or end event
is automatically inferred.
event type
use the “time”, “error”, “receive”, “send”, “notify”, “publish”, “escalate”,
“terminate” keywords
link events derived from the lack of a matching fragment
exclusive gateway split/merge repeat the last common task before the split
labeled exclusive gateway split use “?” after the name of the label
loops repeat the tasks in the loop
sequence flow state tasks in the order in which they happen
conditional flow state the condition following the gateway label
message flow use matching names for the throw and catch events
event-based gateway the last common task before the split has to be followed by events
parallel AND gateway separate the parallel tasks with “|”
lanes state the name of the lane followed by “ :” and a task
pools lanes become pools in case of message exchange
text annotation use “//” before the task to which the text annotation is to be attached
BPMN Sketch Miner October, 2020 26 / 59

27. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Constructs Support
BPMN Constructs Support
Challenges
Trade-off between expressiveness and learnability
Right balance of mining vs modelling
Explicitly modelled
First-class citizen constructs (tasks and lanes)
Parallelism (stating all possible permutations too time consuming)
Refinement constructs (task/event types, labeling XOR gateways
and conditions)
BPMN Construct
Derived
from mining
Explicitly
modeled
DSL construct
tasks state each task name in a new line
task type use the “manual”, “user”, “script”, “rule”, “service”, “send”, “receive” keywords
events
use “( )” to indicate an event. Whether it is a start, intermediate or end event
is automatically inferred.
event type
use the “time”, “error”, “receive”, “send”, “notify”, “publish”, “escalate”,
“terminate” keywords
link events derived from the lack of a matching fragment
exclusive gateway split/merge repeat the last common task before the split
labeled exclusive gateway split use “?” after the name of the label
loops repeat the tasks in the loop
sequence flow state tasks in the order in which they happen
conditional flow state the condition following the gateway label
message flow use matching names for the throw and catch events
event-based gateway the last common task before the split has to be followed by events
parallel AND gateway separate the parallel tasks with “|”
lanes state the name of the lane followed by “ :” and a task
pools lanes become pools in case of message exchange
text annotation use “//” before the task to which the text annotation is to be attached
BPMN Sketch Miner October, 2020 27 / 59

28. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
Context-free grammar
text : := (<trace> ’EOL’)+ ’EOF’;
trace : := <dots> (<line> ’EOL’ <dots>)+ ;
dots : := [’...’ ’EOL’];
line : := <parallel> | <annotation> | <comment>;
comment : := ’///’ (any character until EOL)∗ ;
annotation : := ’//’ <label>;
parallel : := <element> (’|’ <element> )∗ ;
element : := [<role>] (<task> | <event> | <XOR label>);
role : := <label> ’ :’;
XOR label : := <label> ’?’;
task : := [<ttype>] <label>;
event : := ’(’ [<etype>] <label> ’)’;
ttype : := (’send’ | ’receive’ | ’user’ | ’manual’ | ’service’ | ’script’
| ’rule’ );
etype : := (’start’ | ’finish’ | ’timer’| ’send’ | ’receive’ | ’publish’
| ’notify’ | ’error’ | ’escalate’ | ’terminate’ );
label : := any valid BPMN element label;
Simplified Inventory Process Example
BPMN Sketch Miner October, 2020 28 / 59

29. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
BPMN Sketch Miner October, 2020 29 / 59

30. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
BPMN Sketch Miner October, 2020 30 / 59

31. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
BPMN Sketch Miner October, 2020 31 / 59

32. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
BPMN Sketch Miner October, 2020 32 / 59

33. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
BPMN Sketch Miner October, 2020 33 / 59

34. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
BPMN Sketch Miner October, 2020 34 / 59

35. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Textual DSL
BPMN Sketch Miner October, 2020 35 / 59

36. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
DSL Design Decisions
Model Generation and Transformation Pipeline
Renderer
Automatic
Layout
Traces
BPMN Model
(Control flow)
DSL Parser
Textual BPMN
Process Mining
Algorithm
Model
Tranformation
PNG Image
SVG Image
BPMN Model
(with Diagram
Interchange
metadata)
Role, Task and
Event Annotations
BPMN Model
(Control Flow, Pools,
Swimlanes, Events)
BPMN Sketch Miner October, 2020 36 / 59

37. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation Design
Evaluation Design
Research Questions :
RQ1 : Does BPMN Sketch Miner help BPMN non-experts to produce accurate BPMN models in a relatively
short time?
RQ2 : What is the perceived usability of the tool and its features?
BPMN Sketch Miner October, 2020 37 / 59

38. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation Design
Evaluation Design
Research Questions :
RQ1 : Does BPMN Sketch Miner help BPMN non-experts to produce accurate BPMN models in a relatively
short time?
RQ2 : What is the perceived usability of the tool and its features?
Evaluation Design :
BPMN Sketch Miner October, 2020 38 / 59

39. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation Design
Evaluation Design
Research Questions :
RQ1 : Does BPMN Sketch Miner help BPMN non-experts to produce accurate BPMN models in a relatively
short time?
RQ2 : What is the perceived usability of the tool and its features?
Evaluation Design :
BPMN Sketch Miner October, 2020 39 / 59

40. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation Design
Evaluation Design : students participation
BPMN Sketch Miner October, 2020 40 / 59

41. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation Design
Modelling Exercise
BPMN Sketch Miner October, 2020 41 / 59

42. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Modelling accuracy
Modelling accuracy
Model Accuracy vs. Time for solutions with different orders of process instance :
70% 75% 80% 85% 90% 95% 100%
0:00
0:30
1:00
1:30
Accuracy
Time(hh:mm)
12345 12435 12354 21345 51234 51243 51324 51423
⇒ Only 38% of the students enumerated the instances in the order in which they appeared in the process description
BPMN Sketch Miner October, 2020 42 / 59

43. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Modelling accuracy
Modelling accuracy
Model Accuracy vs. Time for solutions with different orders of process instance :
70% 75% 80% 85% 90% 95% 100%
0:00
0:30
1:00
1:30
Accuracy
Time(hh:mm)
12345 12435 12354 21345 51234 51243 51324 51423
⇒ Only 38% of the students enumerated the instances in the order in which they appeared in the process description.
⇒ Students were able to produce an accurate BPMN representation of the model, with an average correctness grade of 87%.
BPMN Sketch Miner October, 2020 43 / 59

44. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Modelling accuracy
Modelling accuracy
Model Accuracy vs. Time for solutions with different orders of process instance :
70% 75% 80% 85% 90% 95% 100%
0:00
0:30
1:00
1:30
Accuracy
Time(hh:mm)
12345 12435 12354 21345 51234 51243 51324 51423
⇒ Only 38% of the students enumerated the instances in the order in which they appeared in the process description.
⇒ Students were able to produce an accurate BPMN representation of the model, with an average correctness grade of 87%.
⇒ All study participants finished the task in less than an hour and a half, with most of them taking between half an hour and an hour.
BPMN Sketch Miner October, 2020 44 / 59

45. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Usage tracking
Tutorials and documentation visits
Engagement (Modeling time) and Retention (Number of visited days) vs. Onboarding (Number of visited tutorials and examples) for Students
and Analysts :
0 10 20 30 40 50 60 70 80
0:00
0:30
1:00
1:30
2:00
Visited Tutorials and Examples
ModelingTime(hours)
1 day (177)
2 days (69)
3 days (23)
4 days (12)
5 days (3)
6 days (2)
7 days (4)
8 days (0)
9 days (2)
10+ days (3)
⇒ Correlations : Visits/Time (0.60), Time/Days (0.73), Visits/Days (0.60)
⇒ 40% of the users did return to the website at least once during the observed period
BPMN Sketch Miner October, 2020 45 / 59

46. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Constructs usage
DSL Expressiveness : Constructs usage
Construct usage vs. Accuracy (multiple constructs can share the same line) :
74 74 78 79 79 82 83 84 85 85 88 88 89 89 89 94 96 97 97 99 100
0
50
100
150
200
Accuracy (each bar represents the model of each participant)
NumberofLines
Empty Lines
Comments
AND Gateways
Fragments
XOR Labels
Events
Tasks
Roles
⇒ 48% of the students used constructs which were not explained in the training.
BPMN Sketch Miner October, 2020 46 / 59

47. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
Evaluation design : analysts participation
BPMN Sketch Miner October, 2020 47 / 59

48. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
Usability study
System Usability Scaling score calculation :
0 10 20 30
NQ1. I found the system unnecessarily complex
NQ2. I think that I would need the support of a technical person to be able to use
this system
NQ3. I thought there was too much inconsistency in this system
NQ4. I found the system very cumbersome to use
NQ5. I needed to learn a lot of things before I could get going with this system
0 10 20 30
PQ1. I think that I would like to use this system frequently
PQ2. I found the various functions in this system were well integrated
PQ3. I would imagine that most people would learn to use this system very quickly
PQ4. I felt very confident using the system
PQ5. I thought the system was easy to use
Strongly disagree Disagree Neutral Agree Strongly agree
BPMN Sketch Miner October, 2020 48 / 59

49. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
Usability study
System Usability Scaling score calculation :
0 10 20 30
NQ1. I found the system unnecessarily complex
NQ2. I think that I would need the support of a technical person to be able to use
this system
NQ3. I thought there was too much inconsistency in this system
NQ4. I found the system very cumbersome to use
NQ5. I needed to learn a lot of things before I could get going with this system
0 10 20 30
PQ1. I think that I would like to use this system frequently
PQ2. I found the various functions in this system were well integrated
PQ3. I would imagine that most people would learn to use this system very quickly
PQ4. I felt very confident using the system
PQ5. I thought the system was easy to use
Strongly disagree Disagree Neutral Agree Strongly agree
–
⇒ A score of 69 as an average of all the answers, with a 95% confidence interval of 64 to 73.
BPMN Sketch Miner October, 2020 49 / 59

50. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
Efficiency and Learnability assessment
System Efficiency and Learnability assessment :
0 10 20 30
NQ1. I found the system unnecessarily complex
NQ2. I think that I would need the support of a technical person to be able to use
this system
NQ3. I thought there was too much inconsistency in this system
NQ4. I found the system very cumbersome to use
NQ5. I needed to learn a lot of things before I could get going with this system
0 10 20 30
PQ1. I think that I would like to use this system frequently
PQ2. I found the various functions in this system were well integrated
PQ3. I would imagine that most people would learn to use this system very quickly
PQ4. I felt very confident using the system
PQ5. I thought the system was easy to use
Strongly disagree Disagree Neutral Agree Strongly agree
–
⇒ Only 2 disagree with the fact that BPMN Sketch Miner is easy to learn.
⇒ Only 1 said he would need support for using the system.
BPMN Sketch Miner October, 2020 50 / 59

51. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
Efficiency and Learnability assessment
System Efficiency and Learnability assessment in the case of the Analysts :
0 2 4 6 8
QA1. I can type faster than I can visually model BPMN diagrams
QA2. Once the processes get large it is hard to navigate through the tex-
tual model
QA3. Once the processes get large it is hard to navigate through the
BPMN visual diagram
QA4. Learning to use the BPMN Sketch Miner was hard for me
QA5. The BPMN Sketch Miner language syntax is easy to remember
Number of analysts
BPMN Sketch Miner October, 2020 51 / 59

52. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
Efficiency and Learnability assessment
System Efficiency and Learnability assessment in the case of the Students :
0 5 10 15 20
QS1. I can type faster than I can visually model BPMN diagrams
QS2. Once the processes get large it is hard to navigate through the tex-
tual model
QS3. Once the processes get large it is hard to navigate through the
BPMN visual diagram
QS4. Learning to use the BPMN Sketch Miner was hard for me
QS5. The BPMN Sketch Miner language syntax is easy to remember
QS6. The generated visual model helped me understand the flow better
QS7. I could easily translate the provided stories into the text describing
the visual model
Number of students
Strongly disagree Disagree Neutral Agree Strongly agree
BPMN Sketch Miner October, 2020 52 / 59

53. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
User Satisfaction
User satisfaction survey results :
Some questions :
How would you rate the BPMN Sketch Miner in
general?
Would the use of the BPMN Sketch Miner in your job
enable you to accomplish tasks more quickly?
Would the BPMN Sketch Miner make it easier to do
your job?
How likely is it that you would recommend the BPMN
Sketch Miner to a friend or colleague?
BPMN Sketch Miner October, 2020 53 / 59

54. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Evaluation design
Quotes from BPMN Sketch Miner users
“Once Upon A time John O’Connell began Staffware and invented BPM. It was really challenging because you had to build workflows in a text file.
Then Staffware invented iProcess modeler which set the BPM world in motion. Graphically building workflows that did real work. Intuitive, awesome
for businesses to build workflows and IT to implement them.
Then everyone saw it was good and made their own.
Then everyone wanted more capabilities and a standard for models BPMN and BPEL was born.
The modelers grew too complicated for business owners to easily build workflows, so it was back to writing specs and having IT build stuff.
Something had to be done.
BPMN-Sketch-Miner came along and fixed the problem by making it quick and easy to build a workflow by typing text again, and generating
visual workflows.”
“Great idea, easy to use. Amazing integration of BPMN objects.”
“Fantastic tool, having daily use of it. Link feature is impressive.”
“It is a really useful tool for learning to create the model without taking care of positioning. Also it is easy to learn the basic concepts.”
“it is a very intuitive tool that I will use from now on and recommend to my work colleagues.”
— BPMN Sketch Miner users
BPMN Sketch Miner October, 2020 54 / 59

55. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Conclusions
BPMN Sketch Miner :
Combines modelling with mining.
Minimal input.
Constrained natural language.
A DSL based on an intuitive concrete syntax.
Evaluation :
RQ1 : Does BPMN Sketch Miner help BPMN non-experts to produce accurate BPMN models in a relatively
short time?
RQ2 : What is the perceived usability of the tool and its features?
BPMN Sketch Miner October, 2020 55 / 59

56. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Conclusions
Research Question :
RQ1 : Does BPMN Sketch Miner help BPMN non-experts to produce accurate BPMN models in a relatively
short time?
Evaluation
results :
Evaluation results :
The accuracy of their models of a
non-trivial process with an average
score of 87%.
Everybody managed to finish the task in
less than 1h30.
BPMN Sketch Miner October, 2020 56 / 59

57. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Conclusions
Research Question :
RQ2 : What is the perceived usability of the tool and its features?
Evaluation results :
SUS score of 64 ± 5.
62% found the tool promising, and 19% found it very
powerful.
The trade-off between expressiveness and usability is well
balanced.
SUS score from industry practitioners of 76 ± 7.
The DSL allows expressing the majority of the crucial
constructs of BPMN.
BPMN Sketch Miner October, 2020 57 / 59

58. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
Future Work
Improve the editor with the most voted concepts by the survey respondents : allowing for manual
adjustments in the layout, renaming tasks, and providing model verification.
Provide support of the most requested BPMN constructs (e.g : subprocesses).
Conduct experiments with more complex processes, with higher number of participants, and taking into
account comparing the textual based modeling with graphical modelling tools.
BPMN Sketch Miner October, 2020 58 / 59

59. Introduction BPMN Sketch Miner Design Evaluation Conclusions & Future work
BPMN Sketch Miner
BPMN Sketch Miner : https://www.bpmn-sketch-miner.ai/
Find more details in : “Ana Ivanchikj, Souhaila Serbout, Cesare Pautasso From Text to Visual BPMN Process
Models : Design and Evaluation, Proc. of the 23rd International Conference on Model Driven Engineering
Languages and Systems (MODELS), Montreal, Canada, October 2020”
BPMN Sketch Miner October, 2020 59 / 59