Usability evaluation of DSLs (PhD defence)

Usability evaluation of
Domain-Specific Languages
Ankica Barišić
supervisors: Vasco Amaral, Miguel Goulão
December, 2017

Language
A language is a means of communication
• The user interface is a realization of a language
• A language is a model that describes the allowed terms and how to
compose them into valid sentences
2 Usability evaluation of Domain-Specific LanguagesAnkica Barišić December 2017

Domain-Specific Language (DSL)
3 December 2017Usability evaluation of Domain-Specific LanguagesAnkica Barišić
• Close the gap between problem and solution
domains
• Reduce the need to use computation concepts
• Focus on the domain concepts

DSL benefits
• Productivity gains
• Better time to market
• Avoid error-prone mappings between domain and
software development concepts
• Leverage the expertise of domain experts through
higher abstraction level
• Narrow the design space

Usability evaluation of Domain-Specific LanguagesAnkica Barišić December 20175
[Mernik, 2005]
DSL Lifecycle

DSL Verification
Did I build the
DSL right?
[Mernik, 2005]

DSL Validation
Did I build
the right
DSL???
7
Did I build the
DSL right?
[Mernik, 2005]

Quality in Use i.e. Usability
[ISO/IEC 25010, 2011]

Research Goal
How to systematically engineer Usability in
Domain-Specific Languages (DSLs)?
Thesis objective:
Promote quality in use of DSLs by building a conceptual
framework that supports their development process,
leveraging usability to a first-class concern
9

Research Questions
RQ1: How are we able to model the DSL usability
evaluation?
RQ2: How can we promote usability concerns since an early
stage of development of the DSL?
RQ3: How can we integrate the proposed methodological
approach to build usability evaluation into the development
process of the DSL?
10

Research method
[Wieringa, 2009]

Understand evaluation model
Pheasant
12
Ankica Barišić, Vasco Amaral, Miguel Goulão, Bruno Barroca.
2011. Quality in Use of Domain-Specific Language: a Case
Study. InProceedings of the Workshop on Evaluation and
Usability of Programming Languages and Tools (PLATEAU) at
SPLASH 2011, Portland, Oregon, USA, ACM, October 2011.
DOI: 10.1145/2089155.2089170

DSL development experience
RPG Games
13
Bruno Barroca, Eduardo Marques, Valter Balegas, Vasco
Amaral, Ankica Barišić. 2012. The RPG DSL: a case study of
language engineering using MDD for Generating RPG Games
for Mobile Phones, InProceedings of the 12th Workshop on
Domain-Specific Modeling at SPLASH 2012, Tucson, Arizona,
ACM, October 2012. DOI: 10.1145/2420918.2420923

Usability evaluation modelling
2012. Evaluating the Usability of Domain-Specific Language.
InBook: Formal and Practical Aspects of Domain-Specific
Languages: Recent Developments, edited by Marjan Mernik,
IGI Global, September 2012, pages: 386-407.
DOI: 10.4018/978-1-4666-2092-6
2011. Quality in Use of DSLs: Current Evaluation Methods.
InProceedings of the INFORUM'2011, Coimbra, Portugal,
September, 2011. http://doi.org/10.5281/zenodo.889940

Patterns for usability evaluation
Ankica Barišić, Vasco Amaral, Miguel Goulão, Bruno Barroca. How to
reach a usable DSL? Moving toward a Systematic Evaluation.
InProceedings of the 5th International Workshop on Multi-Paradigm
Modeling (MPM'2011) at Models 2011, Wellington, New Zealand, EASST
Journal, October, 2011. DOI: 10.14279/tuj.eceasst.50.741
Ankica Barišić, Pedro Monteiro, Vasco Amaral, Miguel Goulão, Miguel
Monteiro. 2012. Patterns for Evaluating Usability of Domain-Specific
Languages. InProceedings of the 19th Conference on pattern languages of
programs (PLoP), SPLASH 2012 Tucson, Arizona, USA, October 2012.
http://doi.org/10.5281/zenodo.889927

Agile DSL development
Ankica Barišić, Vasco Amaral, Miguel Goulão, Ademar Aguiar.
2014. Introducing usability concerns early in the DSL
development cycle: FlowSL experience report, InProceedings
of the 1st International Workshop on Model-Driven
Development Processes and Practices at the 17th
International MoDELS Conference, Valencia, Spain, October,
2014, Zenodo. http://doi.org/10.5281/zenodo.889685
FlowSL

Semi-automated evaluation
Ankica Barišić. 2016. STSM Report: Evaluating the efficiency
in use of search-based automated model merge technique",
Multi-Paradigm Modelling for Cyber-Physical Systems
(MPM4CPS). COST action IC1404. European cooperation in
science and technology (COST).
http://doi.org/10.5281/zenodo.237420
DSE Merge

Usability-driven DSL evolution
Ankica Barišić, Vasco Amaral, Miguel Goulão. 2018.
Leveraging teens feedback in the development of a
Domain-Specific Language: the case of programming
low-cost robots. InProceeding of The 33rd
ACM/SIGAPP Symposium On Applied Computing,
Pau, France, April, 2018.
Visualino

Usability Software Engineering - Modeling Environment
(USE-ME)
Ankica Barišić, Vasco Amaral, Miguel Goulão. 2017. Usability Driven
DSL development with USE-ME. Computer Languages, Systems &
Structures. https://doi.org/10.1016/j.cl.2017.06.005
Ankica Barišić. 2017. Framework Support for Usability Evaluation of
Domain-Specific Languages. In Proceedings of SPLASH ’17
Companion, Vancouver, BC, Canada, October 23–27, 2017,
https://doi.org/10.1145/3135932.3135953
Impact Factor:
1.615

Usability evaluation of Domain-Specific LanguagesAnkica Barišić
(USE-ME)
20 December 2017

USE-ME architecture
21 December 2017
Evaluation of UML diagram
specification with a members
of NOVA-LINCS research
centar.
Evaluation of USE-ME prototype tool
with a novice language engineers in
four DSL development projects.

(USE-ME)
22 December 2017

Context Modelling Activity
23 December 2017

Usability evaluation of Domain-Specific LanguagesAnkica Barišić24 December 2017
User Profile classification
WHO?

Context Environment definition
WHERE?

Workflow definition
HOW?

USE-ME Activity diagram
27 December 2017

USE-ME Goal Modeling
WHY?

29 December 2017

30 December 2017

31 December 2017

Research method
Ankica Barišić, Dominique Blouin, Vasco Amaral, Miguel
Goulão. 2017. A Requirements Engineering Approach for
Usability-Driven DSL Development. InProceedings of SLE’17,
Vancouver, BC, Canada, October 23–24, 2017, DOI:
10.1145/3136014.3136027
RDAL USE-ME

USE-MEinDSLlifecycle

DefiningContextandGoals

ExtendingContextandGoals
withdetailsneededfor
Evaluation

SpecifyingtheEvaluation
SurveysandInteractions

ExecutingEvaluationand
creatingaReport

CalculatingSuccessCoverage

Evaluation survey
goo.gl/forms/
Js4Nh8V6VCZvmsAB2
youtu.be/RjIGFex-zQM

Survey participants

Main Contributions
RQ1 - How are we able to model the DSL usability
evaluation?
❏ To address the problem of the absence of the systematic approach
for the DSL usability evaluation we proposed a conceptual
modelling framework called The Usability Software Engineering
Modelling Environment (USE-ME)
42

Main Contributions
RQ1 - How are we able to model the DSL usability
evaluation?
❏ To address the problem of the absence of the systematic approach
for the DSL usability evaluation we proposed a conceptual
modelling framework called The Usability Software Engineering
Modelling Environment (USE-ME)
43
Does the USE-ME conceptual framework enable
End Users to model DSL usability evaluations?

Main Contributions
RQ2 - How can we promote usability concerns since an
early stage of development of the DSL?
❏ To promote usability concerns since an early stage of development
of DSLs, we proposed a set of patterns and specified an
iterative process which integrates well with the DSL
development phases
44

Main Contributions
RQ2 - How can we promote usability concerns since an
early stage of development of the DSL?
❏ To promote usability concerns since an early stage of development
of DSLs, we proposed a set of patterns and specified an
iterative process which integrates well with the DSL
development phases
45
Does the USE-ME conceptual framework enable
End Users in promoting usability concerns since
an early stage of DSL development?

Main Contributions
RQ3 - How can we integrate the proposed
methodological approach to build usability evaluation
into development process of the DSL?
❏ To integrate existing IDE support for development of DSLs with high
Quality in Use, we developed a functional tool prototype
46

Main Contributions
RQ3 - How can we integrate the proposed
methodological approach to build usability evaluation
into development process of the DSL?
❏ To integrate existing IDE support for development of DSLs with high
Quality in Use, we developed a functional tool prototype
47
Does the USE-ME tool support enable End Users
to built usability evaluation into the development
process of the DSL?

Future Work
Improving the usability of approach support
● Evolution of the tool
○ implementation of guiding wizard
○ implementation of verification of rules and goal coverage engine
● Development of usability catalogue
● Identification and illustration of integration points
○ requirements for DSL construction
○ integration with existing approaches from related work (e.g.
collaborative concrete syntax specification)
○ integration with a survey and interaction modeling supports

Acknowledgments
● FCT/MEC NOVA LINCS; PEst UID/ CEC/04516/ 2013
● DSML4MA TUBITAK/0008/2014
● COST Action IC1404 MPM4CPS
● MOVERCADO; FEUP
● CITI/FCT/UNL; PEst-OE/EEI/UI0527/2011

Usability Software Engineering - Modeling Environment (USE-ME)Ankica Barišić
USE-ME Goal Modeling
50

Context Model Class diagram

Context Modeling Activity

Survey results
● Q1: Does the USE-ME conceptual framework enable End Users to
model DSL usability evaluations?
● Q2: Does the USE-ME conceptual framework enable End Users in
promoting usability concerns since an early stage of DSL
development?
● Q3: Does the USE-ME tool support enable End Users to built usability
evaluation into the development process of the DSL?

USE-ME Utility
57 December 2017

Evaluation survey
MATERIALS:
1. Video presentation of
USE-ME
2. Article about
USE-ME approach
3. Prototype tool
4. Background
questionnaire
5. Feedback
questionnaire
58
Jul 1
Jul 15
Jul 25
Aug 10
Aug 25
Sep 12
Sep 20
Sep 30
OPTIONAL
OPTIONAL
Pilot run of survey
Interview run /
Survey update
Survey run
Result
analysisgoo.gl/forms/
Js4Nh8V6VCZvmsAB2
youtu.be/RjIGFex-zQM
Survey
report

Case studies
Visualino
RPG Games FlowSLPheasant
DSE Merge
59 December 2017

Case studies
Visualino
DSE Merge
60 December 2017

Case studies
Visualino
DSE Merge
61 December 2017

Case studies
62 Usability evaluation of Domain-Specific LanguagesAnkica Barišić
Visualino
DSE Merge
December 2017

63
Barišić, Monteiro, Amaral, Goulão, Monteiro:
"Patterns for Evaluating Usability of Domain-Specific
Languages“, InProceedings of the 19th Conference
on pattern languages of programs (PLoP), SPLASH
2012 Tucson, Arizona, USA, October 2012

64
Languages“, InProceedings of the 19th Conference

65
Languages“, In Proceedings of the 19th Conference

67
Query solution in
C++/BEE
(pseudo code based on real code)
Query solution in
Pheasant

Errorrates
Statistical
meaningfulness
68
Sho
uld
we?
Do
we?
How
can
we?
Life
in
the
tren

Time
Statistical
meaningfulness
69
Sho
uld
we?
Do
we?
How
can
we?
Life
in
the
tren
Belgrade2013Barišić:UsabilityevaluationofDSLs

Selfassessment
(0-5)
Statistical
meaningfulness
70
Sho
uld
we?
Do
we?
How
can
we?
Life
in
the
tren
Mean confidence / query
Non -
Inf
C++/BEE 1,04
Pheasant 4,75
Inf
C++/BEE 4,88
Pheasant 4,83

71
MVC1- developed as a proof-of-concept to validate the key
underlying principles and domain concepts
MVC2 - developed in the form of a platform easily customizable by
experts and extensible by developers of the organization's team.

● Usability requirements: Understandability, Readability, Expressiveness,
Efficiency
● Participants:
● Participant 1: Domain Expert (DSL Development), Campaign Manager
(Organization), Product Owner (Agile processes)
● Participant 2: DSL Developer (DSL Development), System Manager
(Organization), Scrum Master (Agile processes)
● Participant 3: DSL Developer (DSL Development), System Manager
(Organization), Scrum Team (Agile processes)
● Participant4: Domain Expert (DSL development), Flow Manager (Organization),
Product Owner (Agile processes)
● Intervention type: Interview
● Results:
● Understandability and readability is much higher than previously
● User is able to express all the concepts
● Editor is still to complex to be used by Campaign Manager – there is a need for
having more simple view to support his needs
● Interaction improvements with language interface, its integration to platform
environment and multi user support are pointed as main lack of this editor
72

RQ1 Which approaches are reported for DSL development?
RQ2 How the Domain Analysis of DSL is performed?
RQ2.1 Which techniques were used for Domain Analysis?
RQ2.2 Which stakeholders are involved into Domain Analysis?
RQ3 How the Evaluation of DSL was performed?
RQ3.1 What methods are used for Evaluation?
RQ3.2 When Evaluation was performed?
75

Comparison criteria
● Accuracy of solution
● Time to solve a
problem
● Personal preference
● Age
● Gender
● Programming
experience

Results:
LEGO VISUALINO
CHILDREN
8-18 years
non-programmers
Majority solved
problem
Time (4-10min)
Strong preference
Just few solved
problem
Time (12-15min)
No preference
ADULTS
18-40 years
programmers
Majority solved
problem
Time (2-6min)
Small preference
Majority solved
problem
Time (10-15min)
No preference
GENDER No significant
difference
No significant
difference

Usability evaluation of DSLs (PhD defence)

Recommended

Recommended

More Related Content

Similar to Usability evaluation of DSLs (PhD defence)

Similar to Usability evaluation of DSLs (PhD defence) (20)

Recently uploaded

Recently uploaded (20)

Usability evaluation of DSLs (PhD defence)