Denise Bombonatti, Catarina Gralha, Ana Moreira, João Araújo, Miguel Goulão, "Usability of Requirements Techniques - A Systematic Literature Review", 31st ACM/SIGAPP Symposium on Applied Computing, Requirements Engineering Track, ACM-SAC 2016, Pisa, Italy, April 4-8, 2016. http://dx.doi.org/10.1145/2851613.2851758

1. Usability of Requirements Techniques:
A Systematic Literature Review
Denise Bombonatti¹, Catarina Gralha², Ana Moreira²,
João Araújo², Miguel Goulão²
¹IBM Brazil, ²Universidade NOVA de Lisboa
April 6, 2016

2. 2 April 6, 2016
A systematic literature
review is…
“...a form of secondary study that uses a
well-defined methodology to identify,
analyse and interpret all available evidence
related to a specific research question in
a way that is unbiased and (to a degree)
repeatable” Barbara
Kitchenham
Stuart
Charters

3. 3 April 6, 2016
How is the usability of
requirements engineering
techniques and tools addressed?

4. 4
[ISO/IEC 25000]
April 6, 2016
Usability
Learnability
Understandability
ReadabilityWritability
Expressiveness

5. 5 April 6, 2016
Search string
("requirements engineering" OR "requirements
specification" OR "requirements model*" OR
"requirements tool" OR "requirements process" OR
"requirements analysis") AND usability AND
(learnability OR understandability OR
expressiveness OR readability OR writability OR
"cognitive requirement*" OR "cognitive model*")

6. 6 April 6, 2016
Search string
("requirements engineering" OR "requirements
specification" OR "requirements model*" OR
"requirements tool" OR "requirements process" OR
"requirements analysis") AND usability AND
(learnability OR understandability OR
expressiveness OR readability OR writability OR
"cognitive requirement*" OR "cognitive model*")
Notion of requirements
engineering approaches

7. 7 April 6, 2016
Search string
("requirements engineering" OR "requirements
specification" OR "requirements model*" OR
"requirements tool" OR "requirements process" OR
"requirements analysis") AND usability AND
(learnability OR understandability OR
expressiveness OR readability OR writability OR
"cognitive requirement*" OR "cognitive model*")
Usability per se

8. 8 April 6, 2016
Search string
("requirements engineering" OR "requirements
specification" OR "requirements model*" OR
"requirements tool" OR "requirements process" OR
"requirements analysis") AND usability AND
(learnability OR understandability OR
expressiveness OR readability OR writability OR
"cognitive requirement*" OR "cognitive model*")
Quality attributes that
compose usability

9. 9 April 6, 2016
Inclusion
criteria
Exclusion
criteria
Papers published
in REJ
That answer the
research question
Papers that did
not answer the
research question
Papers with the
same content in
different versions
Secondary or
tertiary studies

10. 10 April 6, 2016
Data extraction strategy
Demographic data
● authors
● conference or journal
● year
● Google Scholar citations
● digital library
● approach
● baseline
● publication date
● primary study
● goal
● study type
● vested interest

11. 11 April 6, 2016
Data extraction strategy
Demographic data
Usability approaches
studied
● authors
● conference or journal
● year
● Google Scholar citations
● digital library
● approach
● baseline
● publication date
● primary study
● goal
● study type
● vested interest
● usability attributes
● main results of the
usability evaluation
● impact on the efficiency
● impact on the
effectiveness of the
approach

12. 12 April 6, 2016
Data extraction strategy
Demographic data
Usability approaches
studied
Usability evaluation
● authors
● conference or journal
● year
● Google Scholar citations
● digital library
● approach
● baseline
● publication date
● primary study
● goal
● study type
● vested interest
● usability attributes
● main results of the
usability evaluation
● impact on the efficiency
● impact on the
effectiveness of the
approach
● evaluation method
(process, control group,
type of analysis, validity
threats)
● collected data (academic,
industrial origins,
participants number and
background, which data
was collected, raw data
availability)

13. 13 April 6, 2016
Primary studies selection
19427 62 35
full paper
title &
abstract
search string
Total REJ
1996 - march 2015

14. 14 April 6, 2016
Demographic data

15. 15 April 6, 2016
Analysing the demographic data
Approach
Study type
Vested interest
Experiment
Quantitative
assessment
Qualitative
assessment
Expert opinion Research paper
AI [9] [9] [9] [9]
AWARE [7] [7] [7]
ER [6] [30] [30] [30]
HSO [4] [4] [4]
i*-based [26] [18] [18]
NFR [17] [17] [17]
OO-DFD [12] [12]
Provotype [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [1] [5] [1] [5]
SPS [28] [28] [28]
Text [15] [22] [15] [22] [22]
Use cases [12] [12] [29] [29]
WebSpec [24] [24] [24]
Z [20] [20]

16. 16 April 6, 2016
Covered RE approaches
Approach
Study type
Vested interest
Experiment
Quantitative
assessment
Qualitative
assessment
Expert opinion Research paper
AI [9] [9] [9] [9]
AWARE [7] [7] [7]
ER [6] [30] [30] [30]
HSO [4] [4] [4]
i*-based [26] [18] [18]
NFR [17] [17] [17]
OO-DFD [12] [12]
Provotype [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [1] [5] [1] [5]
SPS [28] [28] [28]
Text [15] [22] [15] [22] [22]
Use cases [12] [12] [29] [29]
WebSpec [24] [24] [24]
Z [20] [20]

17. 17 April 6, 2016
Low number of papers involving UML
Approach
Study type
Vested interest
Experiment
Quantitative
assessment
Qualitative
assessment
Expert opinion Research paper
AI [9] [9] [9] [9]
AWARE [7] [7] [7]
ER [6] [30] [30] [30]
HSO [4] [4] [4]
i*-based [26] [18] [18]
NFR [17] [17] [17]
OO-DFD [12] [12]
Provotype [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [1] [5] [1] [5]
SPS [28] [28] [28]
Text [15] [22] [15] [22] [22]
Use cases [12] [12] [29] [29]
WebSpec [24] [24] [24]
Z [20] [20]

18. 18 April 6, 2016
Low number of papers involving UML
Approach
Study type
Vested interest
Experiment
Quantitative
assessment
Qualitative
assessment
Expert opinion Research paper
AI [9] [9] [9] [9]
AWARE [7] [7] [7]
ER [6] [30] [30] [30]
HSO [4] [4] [4]
i*-based [26] [18] [18]
NFR [17] [17] [17]
OO-DFD [12] [12]
Provotype [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [1] [5] [1] [5]
SPS [28] [28] [28]
Text [15] [22] [15] [22] [22]
Use cases [12] [12] [29] [29]
WebSpec [24] [24] [24]
Z [20] [20]2

19. 19 April 6, 2016
≃16% of the papers are experiments
Approach
Study type
Vested interest
Experiment
Quantitative
assessment
Qualitative
assessment
Expert opinion Research paper
AI [9] [9] [9] [9]
AWARE [7] [7] [7]
ER [6] [30] [30] [30]
HSO [4] [4] [4]
i*-based [26] [18] [18]
NFR [17] [17] [17]
OO-DFD [12] [12]
Provotype [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [1] [5] [1] [5]
SPS [28] [28] [28]
Text [15] [22] [15] [22] [22]
Use cases [12] [12] [29] [29]
WebSpec [24] [24] [24]
Z [20] [20]

20. 20 April 6, 2016
In 74% of the papers, authors are involved in
the evaluated approach
Approach
Study type
Vested interest
Experiment
Quantitative
assessment
Qualitative
assessment
Expert opinion Research paper
AI [9] [9] [9] [9]
AWARE [7] [7] [7]
ER [6] [30] [30] [30]
HSO [4] [4] [4]
i*-based [26] [18] [18]
NFR [17] [17] [17]
OO-DFD [12] [12]
Provotype [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [1] [5] [1] [5]
SPS [28] [28] [28]
Text [15] [22] [15] [22] [22]
Use cases [12] [12] [29] [29]
WebSpec [24] [24] [24]
Z [20] [20]

21. 21 April 6, 2016
26% of the papers were evaluated independently
Approach
Study type
Vested interest
Experiment
Quantitative
assessment
Qualitative
assessment
Expert opinion Research paper
AI [9] [9] [9] [9]
AWARE [7] [7] [7]
ER [6] [30] [30] [30]
HSO [4] [4] [4]
i*-based [26] [18] [18]
NFR [17] [17] [17]
OO-DFD [12] [12]
Provotype [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [1] [5] [1] [5]
SPS [28] [28] [28]
Text [15] [22] [15] [22] [22]
Use cases [12] [12] [29] [29]
WebSpec [24] [24] [24]
Z [20] [20]

22. 22 April 6, 2016
Usability approaches

23. 23 April 6, 2016
Analysing the usability approaches
Approach
Usability attributes
Usability Understandability Learnability Readability Writability Expressiveness
AI [9]
AWARE [7]
ER [30] [6] [6]
HSO [4 ]
i*-based [18] [26]
NFR [17] [17]
OO-DFD [12]
Provotype [10] [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [5] [1]
SPS [28] [28]
Text [15] [22] [22] [15] [22]
Use cases [29] [29] [29]
WebSpec [24]
Z [20] [20]

24. 24 April 6, 2016
Predominance of generic usability concepts
Approach
Usability attributes
Usability Understandability Learnability Readability Writability Expressiveness
AI [9]
AWARE [7]
ER [30] [6] [6]
HSO [4 ]
i*-based [18] [26]
NFR [17] [17]
OO-DFD [12]
Provotype [10] [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [5] [1]
SPS [28] [28]
Text [15] [22] [22] [15] [22]
Use cases [29] [29] [29]
WebSpec [24]
Z [20] [20]

25. 25 April 6, 2016
Papers addressing more specific attributes are
still in minority (≃21%)
Approach
Usability attributes
Usability Understandability Learnability Readability Writability Expressiveness
AI [9]
AWARE [7]
ER [30] [6] [6]
HSO [4 ]
i*-based [18] [26]
NFR [17] [17]
OO-DFD [12]
Provotype [10] [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [5] [1]
SPS [28] [28]
Text [15] [22] [22] [15] [22]
Use cases [29] [29] [29]
WebSpec [24]
Z [20] [20]

26. 26 April 6, 2016
≃60% of the approaches help improving usability
Approach
Usability attributes
Usability Understandability Learnability Readability Writability Expressiveness
AI [9]
AWARE [7]
ER [30] [6] [6]
HSO [4 ]
i*-based [18] [26]
NFR [17] [17]
OO-DFD [12]
Provotype [10] [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [5] [1]
SPS [28] [28]
Text [15] [22] [22] [15] [22]
Use cases [29] [29] [29]
WebSpec [24]
Z [20] [20]

27. 27 April 6, 2016
≃10% of the approaches hurt usability
Approach
Usability attributes
Usability Understandability Learnability Readability Writability Expressiveness
AI [9]
AWARE [7]
ER [30] [6] [6]
HSO [4 ]
i*-based [18] [26]
NFR [17] [17]
OO-DFD [12]
Provotype [10] [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [5] [1]
SPS [28] [28]
Text [15] [22] [22] [15] [22]
Use cases [29] [29] [29]
WebSpec [24]
Z [20] [20]

28. 28 April 6, 2016
≃20% of the results depend on the context in
which the usability is evaluated
Approach
Usability attributes
Usability Understandability Learnability Readability Writability Expressiveness
AI [9]
AWARE [7]
ER [30] [6] [6]
HSO [4 ]
i*-based [18] [26]
NFR [17] [17]
OO-DFD [12]
Provotype [10] [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [5] [1]
SPS [28] [28]
Text [15] [22] [22] [15] [22]
Use cases [29] [29] [29]
WebSpec [24]
Z [20] [20]

29. 29 April 6, 2016
≃10% of the results are inconclusive
Approach
Usability attributes
Usability Understandability Learnability Readability Writability Expressiveness
AI [9]
AWARE [7]
ER [30] [6] [6]
HSO [4 ]
i*-based [18] [26]
NFR [17] [17]
OO-DFD [12]
Provotype [10] [10] [10] [10]
SCTL-MUS [16] [16] [16]
SPL-based [5] [1]
SPS [28] [28]
Text [15] [22] [22] [15] [22]
Use cases [29] [29] [29]
WebSpec [24]
Z [20] [20]

30. 30 April 6, 2016
Usability evaluation

31. 31 April 6, 2016
53% provide a detailed description of the evaluation process
From the analysed papers...

32. 32 April 6, 2016
53% provide a detailed description of the evaluation process
37% use a control group to compare the approach with
From the analysed papers...

33. 33 April 6, 2016
53% provide a detailed description of the evaluation process
37% use a control group to compare the approach with
56% use some form of statistics (descriptive or tests)
From the analysed papers...

34. 34 April 6, 2016
53% provide a detailed description of the evaluation process
37% use a control group to compare the approach with
56% use some form of statistics (descriptive or tests)
74% of the evaluations use academic examples
From the analysed papers...

35. 35 April 6, 2016
53% provide a detailed description of the evaluation process
37% use a control group to compare the approach with
56% use some form of statistics (descriptive or tests)
74% of the evaluations use academic examples
From the analysed papers...
32% make the raw data of their evaluation available

36. 36 April 6, 2016
53% provide a detailed description of the evaluation process
37% use a control group to compare the approach with
56% use some form of statistics (descriptive or tests)
74% of the evaluations use academic examples
From the analysed papers...
32% make the raw data of their evaluation available
Evaluations using students (37%) are slightly more frequent

37. 37 April 6, 2016
53% provide a detailed description of the evaluation process
37% use a control group to compare the approach with
56% use some form of statistics (descriptive or tests)
74% of the evaluations use academic examples
From the analysed papers...
32% make the raw data of their evaluation available
Evaluations using students (37%) are slightly more frequent
Evaluations involving students usually have a much
higher number of participants (≃40)

38. 38 April 6, 2016
Main findings
1
There are relatively few studies concerning
usability of requirements approaches
We expect this kind of studies to become more
abundant in a near future

39. 39 April 6, 2016
Main findings
2
We found a low number of papers involving UML
This may be because UML notations have specific
forums for publication

40. 40 April 6, 2016
Main findings
3
There is a dominance of more generic usability
attributes (e.g., understandability and usability)
The main results are typically about the key
advantages of the approach and open research
challenges

41. 41 April 6, 2016
Main findings
Only a minority of the studies use a control group
to compare the approach with some baseline
There is a positive tendency to make available the
raw data of the analysis
Although students are used more frequently, the
involvement of practitioners follows closely
4

42. External validity
42 April 6, 2016
Validity threats
Internal validity
Different keywords
Selection bias
Interpretation bias
Inter-rater agreement
Second reviewer cross-
checking a sample of the
papers
Only papers from REJ
REJ papers are typically
written by RE experts
Are good representatives of
RE mature work

43. 43 April 6, 2016
Research Opportunities
Production of
independent
evaluations of RE
approaches
enhance the perception
of the maturity of the
approaches
potentially increases
their acceptance by
practitioners

44. 44 April 6, 2016
Conclusions
There is relatively little evidence concerning
the usability of the RE approaches
We found a large variety of approaches
submitted to some form of usability assessment
We expect to find an increasing number of studies
concerned with usability in the near future
The RE community is pushing for evaluations with
professional practitioners, in industrial settings

45. 45 April 6, 2016
Future work
Usability evaluation framework
Open access
repository
For sharing resources and results
Independent evaluation of RE approaches

46. Thank you!
Questions?

47. 47 April 6, 2016
Papers distribution
62 12
12
12
13
13

48. 48 April 6, 2016
Papers 1st round: abstract
35

49. 49 April 6, 2016
Papers distribution
35 7
7
7
7
7

50. 50 April 6, 2016
Papers 2nd round: full paper
19

51. 51 April 6, 2016
Papers 3rd round: double check
19